Nov 02 09:34:42.080475 ip-10-0-128-217 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 09:34:42.459337 ip-10-0-128-217 kubenswrapper[2546]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 09:34:42.459337 ip-10-0-128-217 kubenswrapper[2546]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 09:34:42.459337 ip-10-0-128-217 kubenswrapper[2546]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 09:34:42.459337 ip-10-0-128-217 kubenswrapper[2546]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 09:34:42.463963 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.463808    2546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:42.463963 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.463962    2546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:42.463963 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.463968    2546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:42.463963 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.463972    2546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.463977    2546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.463981    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.463985    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.463989    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.463993    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.463996    2546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464000    2546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464011    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464015    2546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464019    2546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464023    2546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464027    2546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464030    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464035    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464038    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464043    2546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464047    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464051    2546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464055    2546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:42.464200 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464059    2546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464068    2546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464072    2546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464076    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464082    2546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464087    2546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464091    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464096    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464101    2546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464106    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464144    2546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464150    2546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464155    2546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464170    2546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464192    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464197    2546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464201    2546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464208    2546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464257    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:42.464964 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464266    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464272    2546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464276    2546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464281    2546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464285    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464295    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464300    2546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464305    2546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464309    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464313    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464318    2546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464322    2546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464327    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464331    2546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464335    2546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464340    2546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464344    2546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464348    2546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464357    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464362    2546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:42.465508 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464367    2546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464371    2546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464376    2546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464381    2546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464401    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464406    2546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464411    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464416    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464420    2546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464425    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464429    2546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464438    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464442    2546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464447    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464451    2546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464456    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464460    2546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464464    2546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464469    2546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464473    2546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:42.465977 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464477    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464482    2546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464486    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464491    2546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464501    2546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464505    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464509    2546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464514    2546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464518    2546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464523    2546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464527    2546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464531    2546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464536    2546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464541    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464545    2546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464549    2546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464561    2546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464581    2546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464588    2546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:42.466469 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464595    2546 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464599    2546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464604    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464619    2546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.464624    2546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465936    2546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465947    2546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465950    2546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465953    2546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465956    2546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465958    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465961    2546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465966    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465969    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465972    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465974    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465977    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465980    2546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465983    2546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465986    2546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:42.466921 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465988    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465991    2546 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465994    2546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465997    2546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.465999    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466002    2546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466004    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466007    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466009    2546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466013    2546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466018    2546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466022    2546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466026    2546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466028    2546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466031    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466033    2546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466036    2546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466038    2546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466042    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:42.467424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466045    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466047    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466050    2546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466052    2546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466055    2546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466058    2546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466060    2546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466063    2546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466065    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466068    2546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466072    2546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466075    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466078    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466081    2546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466084    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466086    2546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466089    2546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466093    2546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466095    2546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:42.467913 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466098    2546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466100    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466103    2546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466106    2546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466109    2546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466111    2546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466114    2546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466117    2546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466119    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466121    2546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466124    2546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466127    2546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466129    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466132    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466135    2546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466137    2546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466140    2546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466143    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466145    2546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466148    2546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:42.468378 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466150    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466153    2546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466155    2546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466166    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466169    2546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466171    2546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466173    2546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466176    2546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466178    2546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466181    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466183    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466186    2546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466188    2546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466191    2546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466196    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466198    2546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466201    2546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466203    2546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466206    2546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:42.468856 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466208    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466231    2546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466239    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466242    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466245    2546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466248    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466250    2546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466253    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466256    2546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466258    2546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466261    2546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466264    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466266    2546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466269    2546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466863    2546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466871    2546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466880    2546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466883    2546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466886    2546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:42.469325 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466889    2546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466892    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466894    2546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466897    2546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466900    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466902    2546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466905    2546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466907    2546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466911    2546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466914    2546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466917    2546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466919    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466922    2546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466925    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466928    2546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466930    2546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466933    2546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466936    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466939    2546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466942    2546 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:42.469763 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466945    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466948    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466950    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466953    2546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466955    2546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466958    2546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466960    2546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466963    2546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466965    2546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466968    2546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466976    2546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466979    2546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466982    2546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466984    2546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466987    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466989    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466992    2546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466994    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466997    2546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:42.470244 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.466999    2546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467003    2546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467006    2546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467008    2546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467011    2546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467013    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467016    2546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467018    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467021    2546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467023    2546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467026    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467029    2546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467031    2546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467034    2546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467036    2546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467039    2546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467042    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467044    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467047    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467049    2546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:42.470692 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467051    2546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467054    2546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467057    2546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467059    2546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467067    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467070    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467072    2546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467075    2546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467077    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467080    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467082    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467085    2546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467087    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467090    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467092    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467095    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467098    2546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467100    2546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467103    2546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467105    2546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:42.471154 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467108    2546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467111    2546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467113    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467116    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467118    2546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467121    2546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467123    2546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467126    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467128    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467130    2546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467133    2546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467136    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467138    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467142    2546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467146    2546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467150    2546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467153    2546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467162    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:42.471651 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467166    2546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467169    2546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467171    2546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.467174    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.467965    2546 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.467980    2546 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.467992    2546 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468001    2546 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468005    2546 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468008    2546 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468013    2546 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468017    2546 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468020    2546 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468023    2546 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468026    2546 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468030    2546 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468033    2546 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468036    2546 flags.go:64] FLAG: --cgroup-root=""
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468039    2546 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468042    2546 flags.go:64] FLAG: --client-ca-file=""
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468044    2546 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468047    2546 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468054    2546 flags.go:64] FLAG: --cluster-domain=""
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468056    2546 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 09:34:42.472118 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468059    2546 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468062    2546 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468066    2546 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468070    2546 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468073    2546 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468076    2546 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468079    2546 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468082    2546 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468091    2546 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468094    2546 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468097    2546 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468100    2546 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468104    2546 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468107    2546 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468110    2546 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468113    2546 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468116    2546 flags.go:64] FLAG: --enable-server="true"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468119    2546 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468126    2546 flags.go:64] FLAG: --event-burst="100"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468129    2546 flags.go:64] FLAG: --event-qps="50"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468132    2546 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468136    2546 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468138    2546 flags.go:64] FLAG: --eviction-hard=""
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468142    2546 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 09:34:42.472803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468145    2546 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468148    2546 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468151    2546 flags.go:64] FLAG: --eviction-soft=""
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468154    2546 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468157    2546 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468159    2546 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468162    2546 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468165    2546 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468168    2546 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468171    2546 flags.go:64] FLAG: --feature-gates=""
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468174    2546 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468177    2546 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468180    2546 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468183    2546 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468186    2546 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468189    2546 flags.go:64] FLAG: --help="false"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468192    2546 flags.go:64] FLAG: --hostname-override="ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468201    2546 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468204    2546 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468207    2546 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468210    2546 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468234    2546 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468237    2546 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 09:34:42.473444 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468242    2546 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468245    2546 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468248    2546 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468251    2546 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468254    2546 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468257    2546 flags.go:64] FLAG: --kube-reserved=""
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468260    2546 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468263    2546 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468266    2546 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468269    2546 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468271    2546 flags.go:64] FLAG: --lock-file=""
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468274    2546 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468278    2546 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468280    2546 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468286    2546 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468289    2546 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468292    2546 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468295    2546 flags.go:64] FLAG: --logging-format="text"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468298    2546 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468301    2546 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468305    2546 flags.go:64] FLAG: --manifest-url=""
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468307    2546 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468312    2546 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468315    2546 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468318    2546 flags.go:64] FLAG: --max-pods="110"
Nov 02 09:34:42.473997 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468321    2546 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468324    2546 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468333    2546 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468336    2546 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468339    2546 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468344    2546 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468347    2546 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468355    2546 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468358    2546 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468361    2546 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468364    2546 flags.go:64] FLAG: --pod-cidr=""
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468367    2546 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468369    2546 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468372    2546 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468375    2546 flags.go:64] FLAG: --port="10250"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468378    2546 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468381    2546 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-083212a889d37c733"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468384    2546 flags.go:64] FLAG: --qos-reserved=""
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468387    2546 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468390    2546 flags.go:64] FLAG: --register-node="true"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468393    2546 flags.go:64] FLAG: --register-with-taints=""
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468397    2546 flags.go:64] FLAG: --registry-burst="10"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468400    2546 flags.go:64] FLAG: --registry-qps="5"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468403    2546 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468405    2546 flags.go:64] FLAG: --reserved-memory=""
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468409    2546 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 09:34:42.474629 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468412    2546 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468415    2546 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468418    2546 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468421    2546 flags.go:64] FLAG: --runonce="false"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468424    2546 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468427    2546 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468430    2546 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468432    2546 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468435    2546 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468444    2546 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468447    2546 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468451    2546 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468454    2546 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468458    2546 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468461    2546 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468464    2546 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468467    2546 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468469    2546 flags.go:64] FLAG: --system-cgroups=""
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468472    2546 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468477    2546 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468480    2546 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468483    2546 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468489    2546 flags.go:64] FLAG: --tls-min-version=""
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468491    2546 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468494    2546 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 09:34:42.475258 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468497    2546 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468500    2546 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468503    2546 flags.go:64] FLAG: --v="2"
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468507    2546 flags.go:64] FLAG: --version="false"
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468511    2546 flags.go:64] FLAG: --vmodule=""
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468515    2546 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468518    2546 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468680    2546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468684    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468687    2546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468689    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468692    2546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468694    2546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468697    2546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468700    2546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468702    2546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468705    2546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468713    2546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468716    2546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468719    2546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468723    2546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:42.475831 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468725    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468728    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468732    2546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468736    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468739    2546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468742    2546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468744    2546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468747    2546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468749    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468751    2546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468754    2546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468757    2546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468759    2546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468762    2546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468764    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468766    2546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468769    2546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468771    2546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468775    2546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:42.476334 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468778    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468781    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468784    2546 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468786    2546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468789    2546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468791    2546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468794    2546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468796    2546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468799    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468801    2546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468810    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468812    2546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468816    2546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468818    2546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468821    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468823    2546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468826    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468828    2546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468831    2546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468833    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:42.476813 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468835    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468838    2546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468840    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468843    2546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468845    2546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468847    2546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468850    2546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468852    2546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468855    2546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468857    2546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468860    2546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468862    2546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468865    2546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468868    2546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468871    2546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468873    2546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468875    2546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468878    2546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468880    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468883    2546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:42.477301 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468885    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468888    2546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468891    2546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468900    2546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468904    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468906    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468909    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468911    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468913    2546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468916    2546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468918    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468921    2546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468923    2546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468926    2546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468928    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468930    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468933    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468936    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468938    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:42.477772 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468941    2546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468943    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468946    2546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468948    2546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468950    2546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468953    2546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468955    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468958    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468960    2546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468963    2546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468965    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468968    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468970    2546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.468972    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.468999    2546 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469037    2546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:42.478332 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469043    2546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469053    2546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469056    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469059    2546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469061    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469064    2546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469066    2546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469069    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469071    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469074    2546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469076    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469079    2546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469082    2546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469084    2546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469086    2546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469089    2546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469092    2546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469094    2546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469097    2546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469099    2546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:42.478724 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469102    2546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469104    2546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469107    2546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469110    2546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469112    2546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469115    2546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469117    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469120    2546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469122    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469125    2546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469127    2546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469130    2546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469133    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469137    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469146    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469149    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469152    2546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469154    2546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469157    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469159    2546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:42.479184 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469162    2546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469164    2546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469167    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469169    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469172    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469175    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469177    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469179    2546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469182    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469185    2546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469187    2546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469190    2546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469192    2546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469194    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469197    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469199    2546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469202    2546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469205    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469207    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469210    2546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:42.479676 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469230    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469234    2546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469236    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469239    2546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469243    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469247    2546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469250    2546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469259    2546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469262    2546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469265    2546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469267    2546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469270    2546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469273    2546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469276    2546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469278    2546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469281    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469283    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469287    2546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469291    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:42.480140 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469294    2546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469297    2546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469300    2546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469303    2546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469306    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469308    2546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469311    2546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469314    2546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469316    2546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469318    2546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469321    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469324    2546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469326    2546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469329    2546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469333    2546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469337    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469339    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469343    2546 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469347    2546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:42.480601 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469350    2546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469352    2546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469360    2546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469363    2546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469367    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469369    2546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.469372    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.476206    2546 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.476233    2546 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.476249    2546 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.476254    2546 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476311    2546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476316    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476319    2546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476322    2546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476325    2546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476328    2546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476331    2546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476334    2546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476337    2546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:42.481045 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476339    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476342    2546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476345    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476347    2546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476350    2546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476353    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476357    2546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476361    2546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476364    2546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476367    2546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476369    2546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476372    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476374    2546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476377    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476381    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476383    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476386    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476389    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476391    2546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:42.481540 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476394    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476397    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476400    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476403    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476405    2546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476408    2546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476410    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476414    2546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476417    2546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476420    2546 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476423    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476425    2546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476428    2546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476430    2546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476433    2546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476435    2546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476438    2546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476441    2546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476443    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476446    2546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:42.481990 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476448    2546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476451    2546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476453    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476456    2546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476458    2546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476461    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476464    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476466    2546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476469    2546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476472    2546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476474    2546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476477    2546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476479    2546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476483    2546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476485    2546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476488    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476491    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476493    2546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476495    2546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:42.482504 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476498    2546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476500    2546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476503    2546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476505    2546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476508    2546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476510    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476513    2546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476519    2546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476522    2546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476524    2546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476527    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476530    2546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476532    2546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476535    2546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476537    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476539    2546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476542    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476544    2546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476547    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476549    2546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:42.482953 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476552    2546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476555    2546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476558    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476560    2546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476563    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476565    2546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476568    2546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476571    2546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476573    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476576    2546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476578    2546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476580    2546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476583    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476585    2546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476588    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476590    2546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476593    2546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476595    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:42.483424 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476597    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.476625    2546 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476774    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476779    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476782    2546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476784    2546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476787    2546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476790    2546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476792    2546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476795    2546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476797    2546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476800    2546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476802    2546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476805    2546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476807    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476810    2546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:42.483853 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476813    2546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476816    2546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476818    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476821    2546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476824    2546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476827    2546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476829    2546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476832    2546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476835    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476837    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476839    2546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476842    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476844    2546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476847    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476850    2546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476854    2546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476857    2546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476860    2546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476863    2546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:42.484238 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476865    2546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476868    2546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476870    2546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476873    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476875    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476877    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476896    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476900    2546 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476903    2546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476906    2546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476908    2546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476911    2546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476914    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476916    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476919    2546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476927    2546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476930    2546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476932    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476935    2546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476937    2546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:42.484689 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476943    2546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476945    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476948    2546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476950    2546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476953    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476955    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476958    2546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476960    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476962    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476965    2546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476967    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476969    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476972    2546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476975    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476977    2546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476979    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476982    2546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476984    2546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476986    2546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:42.485165 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476989    2546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476992    2546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476994    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476997    2546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.476999    2546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477002    2546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477004    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477006    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477009    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477012    2546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477014    2546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477016    2546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477019    2546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477021    2546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477024    2546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477026    2546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477029    2546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477031    2546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477033    2546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477036    2546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:42.485731 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477038    2546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477041    2546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477043    2546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477045    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477048    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477050    2546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477052    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477055    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477057    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477060    2546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477063    2546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477067    2546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477073    2546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477075    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.477100    2546 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 09:34:42.486203 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477132    2546 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477136    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477139    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477141    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477144    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477146    2546 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477149    2546 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477152    2546 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477154    2546 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477157    2546 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477160    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477162    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477165    2546 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477167    2546 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477169    2546 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477172    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477174    2546 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477177    2546 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477179    2546 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477182    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:42.486643 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477184    2546 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477187    2546 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477189    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477192    2546 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477194    2546 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477197    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477199    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477201    2546 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477204    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477208    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477210    2546 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477235    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477238    2546 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477241    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477243    2546 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477246    2546 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477248    2546 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477251    2546 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477253    2546 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:42.487108 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477255    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477259    2546 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477261    2546 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477264    2546 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477266    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477269    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477271    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477274    2546 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477276    2546 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477279    2546 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477281    2546 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477284    2546 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477286    2546 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477289    2546 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477291    2546 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477293    2546 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477296    2546 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477298    2546 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477301    2546 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477303    2546 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:42.487557 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477306    2546 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477308    2546 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477312    2546 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477315    2546 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477317    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477320    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477322    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477325    2546 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477329    2546 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477333    2546 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477336    2546 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477339    2546 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477342    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477345    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477348    2546 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477351    2546 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477353    2546 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477356    2546 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477358    2546 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:42.488007 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477360    2546 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477363    2546 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477365    2546 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477368    2546 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477370    2546 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477372    2546 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477376    2546 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477378    2546 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477380    2546 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477383    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477385    2546 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477387    2546 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477390    2546 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477393    2546 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477395    2546 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477399    2546 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477402    2546 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477404    2546 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477407    2546 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477409    2546 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:42.488452 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477412    2546 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477416    2546 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477419    2546 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477422    2546 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477425    2546 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477427    2546 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477430    2546 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:42.477432    2546 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.477481    2546 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.479398    2546 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.480310    2546 server.go:1014] "Starting client certificate rotation"
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.480415    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.480446    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.482086    2546 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 09:34:42.488930 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.485340    2546 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 09:34:42.495624 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.495608    2546 log.go:25] "Validated CRI v1 runtime API"
Nov 02 09:34:42.500734 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.500717    2546 log.go:25] "Validated CRI v1 image API"
Nov 02 09:34:42.501616 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.501587    2546 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 09:34:42.504845 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.504827    2546 fs.go:135] Filesystem UUIDs: map[2bab3e04-869d-40e0-abf0-80770041a7f4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d59d5d86-e009-4a82-8154-cfe9092d51b2:/dev/nvme0n1p4]
Nov 02 09:34:42.504900 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.504845    2546 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 09:34:42.510442 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.510322    2546 manager.go:217] Machine: {Timestamp:2026-11-02 09:34:42.50905953 +0000 UTC m=+0.317603721 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100044 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d664e7afd2bf906b0bae40b8bcfbc SystemUUID:ec2d664e-7afd-2bf9-06b0-bae40b8bcfbc BootID:b4c7187c-a3b3-472c-b272-280d026c5336 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e9:6f:79:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e9:6f:79:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:a9:4d:7a:10:09 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 09:34:42.510442 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.510441    2546 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 09:34:42.510552 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.510513    2546 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 09:34:42.513594 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.513554    2546 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 09:34:42.513630 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.513607    2546 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 09:34:42.513744 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.513597    2546 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-217.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 09:34:42.513807 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.513750    2546 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 09:34:42.513807 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.513758    2546 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 09:34:42.513861 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.513813    2546 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 09:34:42.513861 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.513831    2546 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 09:34:42.514585 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.514574    2546 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 09:34:42.517923 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.517911    2546 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 09:34:42.518047 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.518038    2546 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 09:34:42.518075 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.518069    2546 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 09:34:42.518102 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.518079    2546 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 09:34:42.518102 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.518088    2546 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 09:34:42.518102 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.518096    2546 kubelet.go:405] "Adding apiserver pod source"
Nov 02 09:34:42.518177 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.518104    2546 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 09:34:42.518496 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.518484    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 09:34:42.520641 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.520605    2546 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 09:34:42.524866 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.524850    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 09:34:42.525524 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.525512    2546 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 09:34:42.525594 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.525542    2546 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 09:34:42.525698 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.525688    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 09:34:42.525745 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.525703    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 09:34:42.525745 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.525709    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 09:34:42.525745 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.525715    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 09:34:42.525745 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.525721    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 09:34:42.525745 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.525727    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 09:34:42.526397 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.526388    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 09:34:42.526438 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.526398    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 09:34:42.526438 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.526405    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 09:34:42.526438 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.526410    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 09:34:42.526438 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.526419    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 09:34:42.526438 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.526430    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 09:34:42.527584 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.527573    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 09:34:42.527625 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.527587    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 09:34:42.532288 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.532256    2546 server.go:1263] "Started kubelet"
Nov 02 09:34:42.532386 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.532339    2546 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 09:34:42.532711 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.532676    2546 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 09:34:42.532711 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.532707    2546 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-217.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 09:34:42.532803 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.532728    2546 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 09:34:42.533094 ip-10-0-128-217 systemd[1]: Started Kubernetes Kubelet.
Nov 02 09:34:42.534271 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.534154    2546 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 09:34:42.536288 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.534467    2546 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 09:34:42.538302 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.538278    2546 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-217.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 09:34:42.538369 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.538300    2546 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 09:34:42.538476 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.537518    2546 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-217.us-east-2.compute.internal.18e430afa42de56f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-217.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-217.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 09:34:42.532181359 +0000 UTC m=+0.340725550,LastTimestamp:2026-11-02 09:34:42.532181359 +0000 UTC m=+0.340725550,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-217.us-east-2.compute.internal,}"
Nov 02 09:34:42.539722 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.539705    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 09:34:42.540106 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.540085    2546 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 09:34:42.540289 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.540270    2546 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 09:34:42.540476 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.540456    2546 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:34:42.540981 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.540967    2546 factory.go:55] Registering systemd factory
Nov 02 09:34:42.540981 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.540983    2546 factory.go:223] Registration of the systemd container factory successfully
Nov 02 09:34:42.541076 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.541055    2546 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 09:34:42.541117 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.541055    2546 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 09:34:42.541117 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.541093    2546 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 09:34:42.541177 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.541171    2546 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 09:34:42.541206 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.541179    2546 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 09:34:42.541278 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.541268    2546 factory.go:153] Registering CRI-O factory
Nov 02 09:34:42.541315 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.541283    2546 factory.go:223] Registration of the crio container factory successfully
Nov 02 09:34:42.541346 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.541338    2546 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 09:34:42.541377 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.541359    2546 factory.go:103] Registering Raw factory
Nov 02 09:34:42.541404 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.541388    2546 manager.go:1199] Started watching for new ooms in manager
Nov 02 09:34:42.542150 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.542134    2546 manager.go:319] Starting recovery of all containers
Nov 02 09:34:42.549920 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.542591    2546 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-217.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 09:34:42.549920 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.543609    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fqglj"
Nov 02 09:34:42.549920 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.549481    2546 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 09:34:42.551677 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.551653    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fqglj"
Nov 02 09:34:42.552591 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.552468    2546 manager.go:324] Recovery completed
Nov 02 09:34:42.554773 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.554752    2546 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 02 09:34:42.557639 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.557623    2546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:42.559504 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.559488    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:42.559573 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.559527    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:42.559573 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.559543    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:42.560133 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.560118    2546 cpu_manager.go:226] "Starting" policy="none"
Nov 02 09:34:42.560133 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.560125    2546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:42.560133 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.560132    2546 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 09:34:42.560286 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.560150    2546 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 09:34:42.560789 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.560777    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:42.560843 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.560804    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:42.560843 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.560814    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:42.561952 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.561884    2546 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-217.us-east-2.compute.internal.18e430afa5ceda8e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-217.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-217.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-217.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 09:34:42.559507086 +0000 UTC m=+0.368051284,LastTimestamp:2026-11-02 09:34:42.559507086 +0000 UTC m=+0.368051284,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-217.us-east-2.compute.internal,}"
Nov 02 09:34:42.562198 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.562186    2546 policy_none.go:50] "Start"
Nov 02 09:34:42.562245 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.562203    2546 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 09:34:42.562245 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.562230    2546 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 09:34:42.564200 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.564189    2546 policy_none.go:44] "Start"
Nov 02 09:34:42.604098 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.603962    2546 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 09:34:42.642702 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.632588    2546 manager.go:341] "Starting Device Plugin manager"
Nov 02 09:34:42.642702 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.632623    2546 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 09:34:42.642702 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.632632    2546 server.go:87] "Starting device plugin registration server"
Nov 02 09:34:42.642702 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.632900    2546 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 09:34:42.642702 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.632910    2546 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 09:34:42.642702 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.633006    2546 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 09:34:42.642702 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.633083    2546 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 09:34:42.642702 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.633090    2546 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 09:34:42.642702 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.633640    2546 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 09:34:42.642702 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.633691    2546 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:34:42.667091 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.667067    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 09:34:42.668307 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.668291    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 09:34:42.668387 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.668314    2546 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 09:34:42.668387 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.668337    2546 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 09:34:42.668387 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.668375    2546 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 09:34:42.672316 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.672296    2546 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:34:42.733499 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.733456    2546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:42.734257 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.734240    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:42.734329 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.734270    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:42.734329 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.734282    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:42.734329 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.734305    2546 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.745590 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.745577    2546 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.745636 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.745596    2546 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-217.us-east-2.compute.internal\": node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:34:42.764429 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.764406    2546 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:34:42.769459 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.769444    2546 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"]
Nov 02 09:34:42.769510 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.769491    2546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:42.770242 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.770208    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:42.770293 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.770257    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:42.770293 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.770271    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:42.772405 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.772394    2546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:42.772545 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.772531    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.772624 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.772561    2546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:42.773056 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.773040    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:42.773135 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.773069    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:42.773135 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.773085    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:42.773135 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.773104    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:42.773135 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.773122    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:42.773135 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.773131    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:42.775246 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.775230    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.775324 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.775260    2546 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:42.776022 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.776009    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:42.776097 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.776028    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:42.776097 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.776037    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:42.812738 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.812722    2546 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found" node="ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.814662 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.814646    2546 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found" node="ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.843847 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.843824    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7bc16cf3ad519d4529637b03314e6bd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"7bc16cf3ad519d4529637b03314e6bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.843928 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.843850    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bc16cf3ad519d4529637b03314e6bd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"7bc16cf3ad519d4529637b03314e6bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.843928 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.843871    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89488fe67d46e24a4bfb2dbd39ca56e0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"89488fe67d46e24a4bfb2dbd39ca56e0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.865416 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.865398    2546 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:34:42.944544 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.944522    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bc16cf3ad519d4529637b03314e6bd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"7bc16cf3ad519d4529637b03314e6bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.944636 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.944578    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89488fe67d46e24a4bfb2dbd39ca56e0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"89488fe67d46e24a4bfb2dbd39ca56e0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.944636 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.944547    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89488fe67d46e24a4bfb2dbd39ca56e0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"89488fe67d46e24a4bfb2dbd39ca56e0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.944636 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.944612    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bc16cf3ad519d4529637b03314e6bd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"7bc16cf3ad519d4529637b03314e6bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.944771 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.944654    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7bc16cf3ad519d4529637b03314e6bd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"7bc16cf3ad519d4529637b03314e6bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.944771 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:42.944688    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7bc16cf3ad519d4529637b03314e6bd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"7bc16cf3ad519d4529637b03314e6bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:42.965662 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:42.965644    2546 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:34:43.066463 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:43.066401    2546 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:34:43.116589 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:43.116570    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:43.117683 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:43.117667    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:43.167316 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:43.167293    2546 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:34:43.267777 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:43.267745    2546 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:34:43.368366 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:43.368303    2546 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:34:43.468963 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:43.468936    2546 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:34:43.480093 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:43.480073    2546 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 09:34:43.480207 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:43.480190    2546 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 09:34:43.558477 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:43.558442    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 09:29:42 +0000 UTC" deadline="2028-07-13 01:00:38.909545674 +0000 UTC"
Nov 02 09:34:43.558477 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:43.558475    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14847h25m55.351074662s"
Nov 02 09:34:43.569564 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:43.569545    2546 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:34:43.641647 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:43.641594    2546 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:34:43.669920 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:43.669898    2546 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:34:43.721206 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:43.721180    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89488fe67d46e24a4bfb2dbd39ca56e0.slice/crio-ee5f0177d6700e6cbf38f06cb2133bcdddbf0610bf4374dacd37a29d58417b34 WatchSource:0}: Error finding container ee5f0177d6700e6cbf38f06cb2133bcdddbf0610bf4374dacd37a29d58417b34: Status 404 returned error can't find the container with id ee5f0177d6700e6cbf38f06cb2133bcdddbf0610bf4374dacd37a29d58417b34
Nov 02 09:34:43.730958 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:43.730938    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 09:34:43.736402 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:43.736377    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc16cf3ad519d4529637b03314e6bd7.slice/crio-10e53b1d2a57c3d2d9d1e7dd55cd97e105ebbfcd12ffda01461a7d808689db63 WatchSource:0}: Error finding container 10e53b1d2a57c3d2d9d1e7dd55cd97e105ebbfcd12ffda01461a7d808689db63: Status 404 returned error can't find the container with id 10e53b1d2a57c3d2d9d1e7dd55cd97e105ebbfcd12ffda01461a7d808689db63
Nov 02 09:34:43.767532 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:43.767511    2546 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:34:43.840430 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:43.840403    2546 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:43.853076 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:43.853059    2546 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 09:34:43.854848 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:43.854834    2546 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:34:43.863863 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:43.863847    2546 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 09:34:44.000840 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.000770    2546 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:34:44.519181 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.519143    2546 apiserver.go:52] "Watching apiserver"
Nov 02 09:34:44.529731 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.529704    2546 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 09:34:44.530385 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.530343    2546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal","openshift-multus/multus-vkr2t","kube-system/global-pull-secret-syncer-8w4cb","kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-p8bj8","openshift-network-diagnostics/network-check-target-2gvpb","kube-system/konnectivity-agent-h48j7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd","openshift-image-registry/node-ca-hxwln","openshift-multus/multus-additional-cni-plugins-8pbvj","openshift-network-operator/iptables-alerter-r8j97","openshift-ovn-kubernetes/ovnkube-node-px4ck","openshift-cluster-node-tuning-operator/tuned-jxnjv","openshift-dns/node-resolver-r85rg"]
Nov 02 09:34:44.532942 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.532912    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.535035 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.535011    2546 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:34:44.535310 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.535285    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.536807 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.536785    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 09:34:44.536916 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.536858    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 09:34:44.536952 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.536920    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 09:34:44.537016 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.537000    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 09:34:44.537098 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.537084    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kdt4x\""
Nov 02 09:34:44.537195 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.537181    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 09:34:44.537364 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.537348    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:44.537439 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:44.537419    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:34:44.538480 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.538463    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:34:44.538585 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.538521    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 09:34:44.538680 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.538661    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-982vl\""
Nov 02 09:34:44.539908 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.539888    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:34:44.540297 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.540281    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 09:34:44.542015 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.541999    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:44.542143 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:44.542076    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:34:44.544470 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.544451    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:44.544551 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:44.544517    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:34:44.546729 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.546712    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:34:44.548682 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.548661    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:34:44.552437 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.552185    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.554599 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554577    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.554695 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554615    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.554695 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554653    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50bebf56-2ddc-47e0-b8f7-008d202e3af3-tmp\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.554695 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554680    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nf7t\" (UniqueName: \"kubernetes.io/projected/50bebf56-2ddc-47e0-b8f7-008d202e3af3-kube-api-access-9nf7t\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.554852 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554703    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/69f3517a-6eb0-4b5a-a5de-fa56577626be-dbus\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:44.554852 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554730    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:44.554852 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554755    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-hosts-file\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:34:44.554852 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554778    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-kubernetes\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.554852 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554801    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-var-lib-kubelet\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.554852 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554825    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-tuned\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.555124 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554847    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-tmp-dir\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:34:44.555124 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554882    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm85w\" (UniqueName: \"kubernetes.io/projected/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-kube-api-access-wm85w\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:44.555124 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554907    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fqq6\" (UniqueName: \"kubernetes.io/projected/18e3a7af-aafb-4491-b696-95f4858b585a-kube-api-access-5fqq6\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.555124 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554931    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-systemd\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.555124 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554953    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gffwf\" (UniqueName: \"kubernetes.io/projected/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-kube-api-access-gffwf\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:34:44.555124 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.554977    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-cnibin\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.555124 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555030    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-host\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.555124 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555100    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.555568 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555132    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.555568 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555179    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-modprobe-d\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.555568 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555242    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysconfig\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.555568 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555297    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysctl-d\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.555568 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555321    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-sys\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.555568 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555344    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/69f3517a-6eb0-4b5a-a5de-fa56577626be-kubelet-config\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:44.555568 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555368    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-system-cni-dir\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.555568 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555418    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysctl-conf\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.555568 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555445    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-run\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.555568 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555466    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-lib-modules\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.555568 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555488    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:44.555568 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.555530    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.557034 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.557013    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:34:44.559430 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.559414    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.579691 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.579664    2546 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 09:34:44.579781 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.579741    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-llvjf"
Nov 02 09:34:44.590385 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.590358    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-llvjf"
Nov 02 09:34:44.636886 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.636852    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 09:34:44.637435 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.637097    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 09:34:44.637435 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.637138    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:34:44.637435 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.637087    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 09:34:44.637435 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.637274    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 09:34:44.637435 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.637306    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 09:34:44.637435 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.637351    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 09:34:44.637435 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.637369    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t9b8s\""
Nov 02 09:34:44.637435 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.637438    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zc5hs\""
Nov 02 09:34:44.638272 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638250    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 09:34:44.638368 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638288    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 09:34:44.638368 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638301    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 09:34:44.638368 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638319    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 09:34:44.638801 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638570    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 09:34:44.638801 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638648    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7vqld\""
Nov 02 09:34:44.638801 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638662    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 09:34:44.638801 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638684    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 09:34:44.638801 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638698    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qbp4s\""
Nov 02 09:34:44.638801 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638712    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 09:34:44.638801 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638724    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9tqmn\""
Nov 02 09:34:44.638801 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638731    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 09:34:44.638801 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638780    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 09:34:44.638801 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.638785    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 09:34:44.639866 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.639846    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 09:34:44.639989 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.639900    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 09:34:44.639989 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.639944    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-llzhk\""
Nov 02 09:34:44.640086 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.640033    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f2j22\""
Nov 02 09:34:44.642236 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.642206    2546 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 09:34:44.655882 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.655854    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/69f3517a-6eb0-4b5a-a5de-fa56577626be-dbus\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:44.656005 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.655886    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:44.656005 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.655905    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-serviceca\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:34:44.656005 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.655921    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-cni-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.656005 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.655934    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-hostroot\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.656005 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.655950    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-socket-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.656005 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.655968    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-run-ovn-kubernetes\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.656005 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656002    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-tuned\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656027    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-tmp-dir\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656046    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm85w\" (UniqueName: \"kubernetes.io/projected/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-kube-api-access-wm85w\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656060    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-kubelet\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656074    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-etc-kubernetes\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656090    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-systemd\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656113    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gffwf\" (UniqueName: \"kubernetes.io/projected/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-kube-api-access-gffwf\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656130    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-cnibin\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656147    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9sfz\" (UniqueName: \"kubernetes.io/projected/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-kube-api-access-h9sfz\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656165    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef320c69-127d-437a-87ed-cfcd9db66325-agent-certs\") pod \"konnectivity-agent-h48j7\" (UID: \"ef320c69-127d-437a-87ed-cfcd9db66325\") " pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656182    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656204    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-sys-fs\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656239    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-slash\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656253    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-systemd\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656267    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qq5\" (UniqueName: \"kubernetes.io/projected/260f9ad0-a637-4423-ac4c-626c79e6bca9-kube-api-access-r5qq5\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656282    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysconfig\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656296    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/69f3517a-6eb0-4b5a-a5de-fa56577626be-kubelet-config\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:44.656384 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656312    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-system-cni-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656327    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-cni-bin\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656341    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-cni-multus\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656356    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656387    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvstx\" (UniqueName: \"kubernetes.io/projected/cd36afed-d103-4f7a-ba99-00c841bb9537-kube-api-access-rvstx\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656413    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-log-socket\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656435    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-run\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656458    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-lib-modules\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656478    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovn-node-metrics-cert\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656494    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-registration-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656511    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-device-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656526    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5cf1aef2-0bc5-445e-880d-90215888057a-iptables-alerter-script\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656542    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xbhc\" (UniqueName: \"kubernetes.io/projected/5cf1aef2-0bc5-445e-880d-90215888057a-kube-api-access-6xbhc\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656556    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovnkube-config\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656582    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-env-overrides\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656599    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-socket-dir-parent\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.656865 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656624    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-k8s-cni-cncf-io\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656638    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50bebf56-2ddc-47e0-b8f7-008d202e3af3-tmp\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656652    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nf7t\" (UniqueName: \"kubernetes.io/projected/50bebf56-2ddc-47e0-b8f7-008d202e3af3-kube-api-access-9nf7t\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656668    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-hosts-file\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656683    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-etc-selinux\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656697    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cf1aef2-0bc5-445e-880d-90215888057a-host-slash\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656711    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-kubernetes\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656725    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-var-lib-kubelet\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656742    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqq6\" (UniqueName: \"kubernetes.io/projected/18e3a7af-aafb-4491-b696-95f4858b585a-kube-api-access-5fqq6\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656757    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7bjq\" (UniqueName: \"kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq\") pod \"network-check-target-2gvpb\" (UID: \"10db22d0-6805-4d79-a670-e7b96a3b9c22\") " pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656778    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-cnibin\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656793    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-etc-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656807    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-ovn\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656837    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttf7t\" (UniqueName: \"kubernetes.io/projected/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-kube-api-access-ttf7t\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656852    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-run-netns\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656868    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-cni-netd\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656884    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.657347 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656903    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-host\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656918    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656935    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656949    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-cni-binary-copy\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656964    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-conf-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656978    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-cni-bin\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.656993    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-modprobe-d\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657009    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysctl-d\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657028    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-sys\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657042    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-system-cni-dir\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657057    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-netns\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657075    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-multus-certs\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657090    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-node-log\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657105    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysctl-conf\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657120    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657135    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657151    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-daemon-config\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.658026 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657172    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-systemd-units\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657187    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-host\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657202    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovnkube-script-lib\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657233    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657258    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef320c69-127d-437a-87ed-cfcd9db66325-konnectivity-ca\") pod \"konnectivity-agent-h48j7\" (UID: \"ef320c69-127d-437a-87ed-cfcd9db66325\") " pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657285    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-kubelet\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657315    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-var-lib-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.657601    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/69f3517a-6eb0-4b5a-a5de-fa56577626be-dbus\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:44.657669    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:44.657741    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret podName:69f3517a-6eb0-4b5a-a5de-fa56577626be nodeName:}" failed. No retries permitted until 2026-11-02 09:34:45.1577191 +0000 UTC m=+2.966263344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret") pod "global-pull-secret-syncer-8w4cb" (UID: "69f3517a-6eb0-4b5a-a5de-fa56577626be") : object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.658046    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-systemd\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.658267    2546 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.658334    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-cnibin\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.658459    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysconfig\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:44.658544    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.658562    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-kubernetes\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.658582    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-var-lib-kubelet\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.658818 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:44.658597    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs podName:d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa nodeName:}" failed. No retries permitted until 2026-11-02 09:34:45.15858165 +0000 UTC m=+2.967125844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs") pod "network-metrics-daemon-p8bj8" (UID: "d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:44.659659 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.658655    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-sys\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.659659 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.658690    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-system-cni-dir\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.659659 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.658756    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/69f3517a-6eb0-4b5a-a5de-fa56577626be-kubelet-config\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:44.659659 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.658775    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-host\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.659659 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.658869    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-lib-modules\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.659659 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.658869    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-modprobe-d\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.659659 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.658949    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-run\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.659659 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.659148    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysctl-conf\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.659659 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.659512    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.659659 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.659572    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-hosts-file\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:34:44.660136 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.659681    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.660136 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.659770    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysctl-d\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.660136 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.659876    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-tmp-dir\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:34:44.660136 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.660076    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.660482 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.660462    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.662813 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.662787    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-tuned\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.662885 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.662811    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50bebf56-2ddc-47e0-b8f7-008d202e3af3-tmp\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.673493 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.673403    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal" event={"ID":"7bc16cf3ad519d4529637b03314e6bd7","Type":"ContainerStarted","Data":"10e53b1d2a57c3d2d9d1e7dd55cd97e105ebbfcd12ffda01461a7d808689db63"}
Nov 02 09:34:44.674964 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.674940    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nf7t\" (UniqueName: \"kubernetes.io/projected/50bebf56-2ddc-47e0-b8f7-008d202e3af3-kube-api-access-9nf7t\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.675136 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.675111    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm85w\" (UniqueName: \"kubernetes.io/projected/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-kube-api-access-wm85w\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:44.675467 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.675443    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal" event={"ID":"89488fe67d46e24a4bfb2dbd39ca56e0","Type":"ContainerStarted","Data":"ee5f0177d6700e6cbf38f06cb2133bcdddbf0610bf4374dacd37a29d58417b34"}
Nov 02 09:34:44.675672 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.675650    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fqq6\" (UniqueName: \"kubernetes.io/projected/18e3a7af-aafb-4491-b696-95f4858b585a-kube-api-access-5fqq6\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.676489 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.676467    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gffwf\" (UniqueName: \"kubernetes.io/projected/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-kube-api-access-gffwf\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:34:44.757804 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.757772    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-system-cni-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.757804 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.757802    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-cni-bin\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.757991 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.757829    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-cni-multus\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.757991 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.757853    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.757991 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.757877    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvstx\" (UniqueName: \"kubernetes.io/projected/cd36afed-d103-4f7a-ba99-00c841bb9537-kube-api-access-rvstx\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.757991 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.757900    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-log-socket\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.757991 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.757926    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovn-node-metrics-cert\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.757991 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.757953    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-registration-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.757991 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.757970    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-device-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.757991 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.757985    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5cf1aef2-0bc5-445e-880d-90215888057a-iptables-alerter-script\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.757999    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xbhc\" (UniqueName: \"kubernetes.io/projected/5cf1aef2-0bc5-445e-880d-90215888057a-kube-api-access-6xbhc\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758015    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovnkube-config\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758039    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-env-overrides\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758113    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-log-socket\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758162    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-socket-dir-parent\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758192    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-k8s-cni-cncf-io\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758242    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-etc-selinux\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758268    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cf1aef2-0bc5-445e-880d-90215888057a-host-slash\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758297    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bjq\" (UniqueName: \"kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq\") pod \"network-check-target-2gvpb\" (UID: \"10db22d0-6805-4d79-a670-e7b96a3b9c22\") " pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758322    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-cnibin\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758331    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758346    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-etc-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758355    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-device-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758370    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-ovn\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.758388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758392    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttf7t\" (UniqueName: \"kubernetes.io/projected/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-kube-api-access-ttf7t\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758416    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-run-netns\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758440    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-cni-netd\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758466    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758496    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-cni-binary-copy\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758519    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-conf-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758560    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-cni-bin\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758577    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-conf-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758583    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-registration-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758584    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-cnibin\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758596    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-netns\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758622    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-multus-certs\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758632    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-ovn\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758639    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-etc-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758647    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-node-log\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758681    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-run-netns\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758692    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-daemon-config\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758695    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.759116 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758625    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-system-cni-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758717    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-systemd-units\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758722    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-k8s-cni-cncf-io\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758742    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-host\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758750    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-cni-bin\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758764    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovnkube-script-lib\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758790    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef320c69-127d-437a-87ed-cfcd9db66325-konnectivity-ca\") pod \"konnectivity-agent-h48j7\" (UID: \"ef320c69-127d-437a-87ed-cfcd9db66325\") " pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758793    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-cni-multus\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758813    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-kubelet\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758831    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-cni-bin\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758837    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-var-lib-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758868    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-netns\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758873    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-serviceca\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758904    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-multus-certs\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758896    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-cni-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758945    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-hostroot\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758957    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cf1aef2-0bc5-445e-880d-90215888057a-host-slash\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758983    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-env-overrides\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.760019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759004    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-etc-selinux\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759054    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-cni-netd\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759064    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-socket-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759098    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-node-log\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759101    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-run-ovn-kubernetes\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759133    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-run-ovn-kubernetes\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759143    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-kubelet\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759169    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-etc-kubernetes\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759191    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5cf1aef2-0bc5-445e-880d-90215888057a-iptables-alerter-script\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759200    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9sfz\" (UniqueName: \"kubernetes.io/projected/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-kube-api-access-h9sfz\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759250    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-kubelet\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759257    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-cni-binary-copy\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.758691    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-socket-dir-parent\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759283    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef320c69-127d-437a-87ed-cfcd9db66325-agent-certs\") pod \"konnectivity-agent-h48j7\" (UID: \"ef320c69-127d-437a-87ed-cfcd9db66325\") " pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759315    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-etc-kubernetes\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759317    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759334    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759358    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-sys-fs\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.760794 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759383    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-slash\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.761644 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759406    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-systemd\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.761644 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759429    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qq5\" (UniqueName: \"kubernetes.io/projected/260f9ad0-a637-4423-ac4c-626c79e6bca9-kube-api-access-r5qq5\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.761644 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759171    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-host\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:34:44.761644 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759514    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-sys-fs\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.761644 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759541    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-systemd-units\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.761644 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759552    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-slash\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.761644 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759587    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-systemd\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.761644 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759624    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-kubelet\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.761644 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759680    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-daemon-config\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.761644 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759716    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-var-lib-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.761644 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759739    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-hostroot\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.761644 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759774    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-cni-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.761644 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.759854    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-socket-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.764340 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.764317    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-serviceca\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:34:44.764511 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.764492    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef320c69-127d-437a-87ed-cfcd9db66325-konnectivity-ca\") pod \"konnectivity-agent-h48j7\" (UID: \"ef320c69-127d-437a-87ed-cfcd9db66325\") " pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:34:44.764670 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.764649    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovnkube-config\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.764764 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.764747    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovnkube-script-lib\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.764950 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.764933    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovn-node-metrics-cert\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.764985 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.764967    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef320c69-127d-437a-87ed-cfcd9db66325-agent-certs\") pod \"konnectivity-agent-h48j7\" (UID: \"ef320c69-127d-437a-87ed-cfcd9db66325\") " pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:34:44.768322 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:44.768295    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 09:34:44.768322 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:44.768318    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 09:34:44.768476 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:44.768332    2546 projected.go:196] Error preparing data for projected volume kube-api-access-j7bjq for pod openshift-network-diagnostics/network-check-target-2gvpb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:44.768476 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:44.768395    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq podName:10db22d0-6805-4d79-a670-e7b96a3b9c22 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:45.268377442 +0000 UTC m=+3.076921624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j7bjq" (UniqueName: "kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq") pod "network-check-target-2gvpb" (UID: "10db22d0-6805-4d79-a670-e7b96a3b9c22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:44.770702 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.770615    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xbhc\" (UniqueName: \"kubernetes.io/projected/5cf1aef2-0bc5-445e-880d-90215888057a-kube-api-access-6xbhc\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:34:44.771519 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.771497    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttf7t\" (UniqueName: \"kubernetes.io/projected/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-kube-api-access-ttf7t\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.771707 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.771684    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9sfz\" (UniqueName: \"kubernetes.io/projected/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-kube-api-access-h9sfz\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:34:44.772809 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.772783    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvstx\" (UniqueName: \"kubernetes.io/projected/cd36afed-d103-4f7a-ba99-00c841bb9537-kube-api-access-rvstx\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.773205 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.773183    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qq5\" (UniqueName: \"kubernetes.io/projected/260f9ad0-a637-4423-ac4c-626c79e6bca9-kube-api-access-r5qq5\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.863807 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.863772    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:34:44.865959 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.865922    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:34:44.869045 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.868535    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:34:44.873268 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.873120    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:34:44.878519 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:44.877267    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bebf56_2ddc_47e0_b8f7_008d202e3af3.slice/crio-1ad35cd15050964ca820d30a80886295a4dd1e4ea40d673ecbd857591cb6475a WatchSource:0}: Error finding container 1ad35cd15050964ca820d30a80886295a4dd1e4ea40d673ecbd857591cb6475a: Status 404 returned error can't find the container with id 1ad35cd15050964ca820d30a80886295a4dd1e4ea40d673ecbd857591cb6475a
Nov 02 09:34:44.880294 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.880064    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:34:44.887660 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.887638    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vkr2t"
Nov 02 09:34:44.890553 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:44.890525    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e3a7af_aafb_4491_b696_95f4858b585a.slice/crio-5e3d93834a6f3ee2e9983e5648b508e4602ef6f81ecf374a7ec917fff5af183d WatchSource:0}: Error finding container 5e3d93834a6f3ee2e9983e5648b508e4602ef6f81ecf374a7ec917fff5af183d: Status 404 returned error can't find the container with id 5e3d93834a6f3ee2e9983e5648b508e4602ef6f81ecf374a7ec917fff5af183d
Nov 02 09:34:44.891078 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:44.891049    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0556aaf2_dc7e_4e57_a721_4e0ee6cfd923.slice/crio-53fa6636a370d2cb041f3004e6a032772392ba78b6b30b8f49082e4d6a04b5ce WatchSource:0}: Error finding container 53fa6636a370d2cb041f3004e6a032772392ba78b6b30b8f49082e4d6a04b5ce: Status 404 returned error can't find the container with id 53fa6636a370d2cb041f3004e6a032772392ba78b6b30b8f49082e4d6a04b5ce
Nov 02 09:34:44.900178 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:44.900155    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef320c69_127d_437a_87ed_cfcd9db66325.slice/crio-a9f564fac29ec9b24b115967e1954028a16075c906a160c07daff681391c6e8e WatchSource:0}: Error finding container a9f564fac29ec9b24b115967e1954028a16075c906a160c07daff681391c6e8e: Status 404 returned error can't find the container with id a9f564fac29ec9b24b115967e1954028a16075c906a160c07daff681391c6e8e
Nov 02 09:34:44.901100 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:44.901076    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab1d8c2_b3fa_43bc_9009_a0a9c3079eab.slice/crio-c8e16647170d3db67b1a128ab9a6baabae43570fa576409e400e59bb6d236856 WatchSource:0}: Error finding container c8e16647170d3db67b1a128ab9a6baabae43570fa576409e400e59bb6d236856: Status 404 returned error can't find the container with id c8e16647170d3db67b1a128ab9a6baabae43570fa576409e400e59bb6d236856
Nov 02 09:34:44.901980 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.901958    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:34:44.906164 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.906141    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:34:44.913438 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:44.913410    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd36afed_d103_4f7a_ba99_00c841bb9537.slice/crio-1fce18bc9f01536ac7a0103974724a762893f0ddfaf69f8316ef71d73ee5b06f WatchSource:0}: Error finding container 1fce18bc9f01536ac7a0103974724a762893f0ddfaf69f8316ef71d73ee5b06f: Status 404 returned error can't find the container with id 1fce18bc9f01536ac7a0103974724a762893f0ddfaf69f8316ef71d73ee5b06f
Nov 02 09:34:44.917056 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:44.917036    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf1aef2_0bc5_445e_880d_90215888057a.slice/crio-8bcc5a4312ea98f8d8dda276dd1dbe28ee676cb992f5cb49b90a023e42680e73 WatchSource:0}: Error finding container 8bcc5a4312ea98f8d8dda276dd1dbe28ee676cb992f5cb49b90a023e42680e73: Status 404 returned error can't find the container with id 8bcc5a4312ea98f8d8dda276dd1dbe28ee676cb992f5cb49b90a023e42680e73
Nov 02 09:34:44.928337 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:44.928315    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:34:44.940282 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:34:44.940256    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260f9ad0_a637_4423_ac4c_626c79e6bca9.slice/crio-069a9cd2953fdceae912f233fb6c8b15ddeeec78f0099252f640c71624b9e444 WatchSource:0}: Error finding container 069a9cd2953fdceae912f233fb6c8b15ddeeec78f0099252f640c71624b9e444: Status 404 returned error can't find the container with id 069a9cd2953fdceae912f233fb6c8b15ddeeec78f0099252f640c71624b9e444
Nov 02 09:34:45.163055 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.163025    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:45.163178 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.163063    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:45.163178 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:45.163166    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:45.163318 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:45.163193    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:45.163318 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:45.163216    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret podName:69f3517a-6eb0-4b5a-a5de-fa56577626be nodeName:}" failed. No retries permitted until 2026-11-02 09:34:46.163202842 +0000 UTC m=+3.971747036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret") pod "global-pull-secret-syncer-8w4cb" (UID: "69f3517a-6eb0-4b5a-a5de-fa56577626be") : object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:45.163318 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:45.163278    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs podName:d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa nodeName:}" failed. No retries permitted until 2026-11-02 09:34:46.163259558 +0000 UTC m=+3.971803764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs") pod "network-metrics-daemon-p8bj8" (UID: "d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:45.365068 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.365006    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bjq\" (UniqueName: \"kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq\") pod \"network-check-target-2gvpb\" (UID: \"10db22d0-6805-4d79-a670-e7b96a3b9c22\") " pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:45.365217 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:45.365175    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 09:34:45.365217 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:45.365198    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 09:34:45.365217 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:45.365209    2546 projected.go:196] Error preparing data for projected volume kube-api-access-j7bjq for pod openshift-network-diagnostics/network-check-target-2gvpb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:45.365371 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:45.365272    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq podName:10db22d0-6805-4d79-a670-e7b96a3b9c22 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:46.365257592 +0000 UTC m=+4.173801771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j7bjq" (UniqueName: "kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq") pod "network-check-target-2gvpb" (UID: "10db22d0-6805-4d79-a670-e7b96a3b9c22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:45.594818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.594539    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 09:29:44 +0000 UTC" deadline="2028-05-02 03:35:52.123846732 +0000 UTC"
Nov 02 09:34:45.594818 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.594821    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13122h1m6.529033131s"
Nov 02 09:34:45.669547 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.669480    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:45.670091 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:45.670054    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:34:45.677477 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.677416    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:45.677619 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:45.677592    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:34:45.683947 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.683915    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h48j7" event={"ID":"ef320c69-127d-437a-87ed-cfcd9db66325","Type":"ContainerStarted","Data":"a9f564fac29ec9b24b115967e1954028a16075c906a160c07daff681391c6e8e"}
Nov 02 09:34:45.691746 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.691557    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbvj" event={"ID":"18e3a7af-aafb-4491-b696-95f4858b585a","Type":"ContainerStarted","Data":"5e3d93834a6f3ee2e9983e5648b508e4602ef6f81ecf374a7ec917fff5af183d"}
Nov 02 09:34:45.693649 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.693489    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r85rg" event={"ID":"8e9ee3cd-e542-40a1-a18f-2843dcd82a69","Type":"ContainerStarted","Data":"197ec8805706c766b2b8fc78d4600efcd2c70cbbda82317c9109581ed62b3b47"}
Nov 02 09:34:45.696011 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.695950    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jxnjv" event={"ID":"50bebf56-2ddc-47e0-b8f7-008d202e3af3","Type":"ContainerStarted","Data":"1ad35cd15050964ca820d30a80886295a4dd1e4ea40d673ecbd857591cb6475a"}
Nov 02 09:34:45.700993 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.700971    2546 generic.go:356] "Generic (PLEG): container finished" podID="7bc16cf3ad519d4529637b03314e6bd7" containerID="b9d0478422abfeb70f1ac0e4ef1d1d0f50c3d7b7383fa494c19cc92504eeae8e" exitCode=0
Nov 02 09:34:45.702567 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.701239    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal" event={"ID":"7bc16cf3ad519d4529637b03314e6bd7","Type":"ContainerDied","Data":"b9d0478422abfeb70f1ac0e4ef1d1d0f50c3d7b7383fa494c19cc92504eeae8e"}
Nov 02 09:34:45.708276 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.707895    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal" event={"ID":"89488fe67d46e24a4bfb2dbd39ca56e0","Type":"ContainerStarted","Data":"27ca94c5429d79d4708706c4c42cfcdf8edae3c7d2eda17e00f13f1919d82753"}
Nov 02 09:34:45.711708 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.711642    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r8j97" event={"ID":"5cf1aef2-0bc5-445e-880d-90215888057a","Type":"ContainerStarted","Data":"8bcc5a4312ea98f8d8dda276dd1dbe28ee676cb992f5cb49b90a023e42680e73"}
Nov 02 09:34:45.720958 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.720879    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vkr2t" event={"ID":"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab","Type":"ContainerStarted","Data":"c8e16647170d3db67b1a128ab9a6baabae43570fa576409e400e59bb6d236856"}
Nov 02 09:34:45.725363 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.725266    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hxwln" event={"ID":"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923","Type":"ContainerStarted","Data":"53fa6636a370d2cb041f3004e6a032772392ba78b6b30b8f49082e4d6a04b5ce"}
Nov 02 09:34:45.727326 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.727275    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck" event={"ID":"260f9ad0-a637-4423-ac4c-626c79e6bca9","Type":"ContainerStarted","Data":"069a9cd2953fdceae912f233fb6c8b15ddeeec78f0099252f640c71624b9e444"}
Nov 02 09:34:45.731592 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.731553    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd" event={"ID":"cd36afed-d103-4f7a-ba99-00c841bb9537","Type":"ContainerStarted","Data":"1fce18bc9f01536ac7a0103974724a762893f0ddfaf69f8316ef71d73ee5b06f"}
Nov 02 09:34:45.738709 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:45.738575    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal" podStartSLOduration=1.738559008 podStartE2EDuration="1.738559008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:45.738142777 +0000 UTC m=+3.546686982" watchObservedRunningTime="2026-11-02 09:34:45.738559008 +0000 UTC m=+3.547103213"
Nov 02 09:34:46.171795 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:46.171705    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:46.171795 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:46.171766    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:46.172032 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:46.171948    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:46.172032 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:46.172012    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret podName:69f3517a-6eb0-4b5a-a5de-fa56577626be nodeName:}" failed. No retries permitted until 2026-11-02 09:34:48.171991505 +0000 UTC m=+5.980535704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret") pod "global-pull-secret-syncer-8w4cb" (UID: "69f3517a-6eb0-4b5a-a5de-fa56577626be") : object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:46.172524 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:46.172431    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:46.172524 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:46.172485    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs podName:d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa nodeName:}" failed. No retries permitted until 2026-11-02 09:34:48.172470272 +0000 UTC m=+5.981014476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs") pod "network-metrics-daemon-p8bj8" (UID: "d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:46.374576 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:46.373779    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bjq\" (UniqueName: \"kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq\") pod \"network-check-target-2gvpb\" (UID: \"10db22d0-6805-4d79-a670-e7b96a3b9c22\") " pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:46.374576 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:46.374039    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 09:34:46.374576 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:46.374105    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 09:34:46.374576 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:46.374134    2546 projected.go:196] Error preparing data for projected volume kube-api-access-j7bjq for pod openshift-network-diagnostics/network-check-target-2gvpb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:46.374576 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:46.374199    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq podName:10db22d0-6805-4d79-a670-e7b96a3b9c22 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:48.374181263 +0000 UTC m=+6.182725449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j7bjq" (UniqueName: "kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq") pod "network-check-target-2gvpb" (UID: "10db22d0-6805-4d79-a670-e7b96a3b9c22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:46.595096 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:46.594970    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 09:29:44 +0000 UTC" deadline="2028-06-16 07:35:18.603466527 +0000 UTC"
Nov 02 09:34:46.595096 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:46.595045    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14206h0m32.008425576s"
Nov 02 09:34:46.670164 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:46.670130    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:46.670354 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:46.670307    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:34:46.749350 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:46.749311    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal" event={"ID":"7bc16cf3ad519d4529637b03314e6bd7","Type":"ContainerStarted","Data":"2a599f32843847a927e0cdc87b580ddf850cb412b6c34639cd66422f1ff79766"}
Nov 02 09:34:47.670268 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:47.669514    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:47.670268 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:47.669688    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:34:47.670268 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:47.670127    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:47.670268 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:47.670224    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:34:48.189533 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:48.188805    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:48.189533 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:48.188858    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:48.189533 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:48.188995    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:48.189533 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:48.189053    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret podName:69f3517a-6eb0-4b5a-a5de-fa56577626be nodeName:}" failed. No retries permitted until 2026-11-02 09:34:52.189036296 +0000 UTC m=+9.997580494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret") pod "global-pull-secret-syncer-8w4cb" (UID: "69f3517a-6eb0-4b5a-a5de-fa56577626be") : object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:48.189533 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:48.189433    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:48.189533 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:48.189487    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs podName:d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa nodeName:}" failed. No retries permitted until 2026-11-02 09:34:52.189469695 +0000 UTC m=+9.998013877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs") pod "network-metrics-daemon-p8bj8" (UID: "d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:48.390656 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:48.389947    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bjq\" (UniqueName: \"kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq\") pod \"network-check-target-2gvpb\" (UID: \"10db22d0-6805-4d79-a670-e7b96a3b9c22\") " pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:48.390656 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:48.390116    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 09:34:48.390656 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:48.390138    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 09:34:48.390656 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:48.390152    2546 projected.go:196] Error preparing data for projected volume kube-api-access-j7bjq for pod openshift-network-diagnostics/network-check-target-2gvpb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:48.390656 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:48.390213    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq podName:10db22d0-6805-4d79-a670-e7b96a3b9c22 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:52.390195087 +0000 UTC m=+10.198739286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j7bjq" (UniqueName: "kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq") pod "network-check-target-2gvpb" (UID: "10db22d0-6805-4d79-a670-e7b96a3b9c22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:48.671964 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:48.671889    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:48.672480 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:48.672086    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:34:48.773879 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:48.773837    2546 generic.go:356] "Generic (PLEG): container finished" podID="18e3a7af-aafb-4491-b696-95f4858b585a" containerID="c04d27bb4cfc72e4d277593d79f3b2402596b919ec90eb242f951cbd0d9a86a0" exitCode=0
Nov 02 09:34:48.774044 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:48.773900    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbvj" event={"ID":"18e3a7af-aafb-4491-b696-95f4858b585a","Type":"ContainerDied","Data":"c04d27bb4cfc72e4d277593d79f3b2402596b919ec90eb242f951cbd0d9a86a0"}
Nov 02 09:34:48.804512 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:48.802813    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal" podStartSLOduration=4.802792897 podStartE2EDuration="4.802792897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:46.770724619 +0000 UTC m=+4.579268823" watchObservedRunningTime="2026-11-02 09:34:48.802792897 +0000 UTC m=+6.611337103"
Nov 02 09:34:49.669607 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:49.669572    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:49.669799 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:49.669708    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:34:49.670167 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:49.670127    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:49.670309 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:49.670254    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:34:49.780430 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:49.779673    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h48j7" event={"ID":"ef320c69-127d-437a-87ed-cfcd9db66325","Type":"ContainerStarted","Data":"d677e5c3374c376196ade60e912fc45f7b5d0d8a2f83e11070e0d9b0c2a7d7ae"}
Nov 02 09:34:50.671902 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:50.671865    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:50.672069 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:50.672003    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:34:50.784864 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:50.784826    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hxwln" event={"ID":"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923","Type":"ContainerStarted","Data":"0317c2f849ac33d692bf89895398742a3a6a0ec12360dd0ee7803a747c1fe5c1"}
Nov 02 09:34:50.801654 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:50.801566    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h48j7" podStartSLOduration=3.866927455 podStartE2EDuration="7.80154649s" totalImagesPullingTime="3.934619035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:49.79956743 +0000 UTC m=+7.608111635" watchObservedRunningTime="2026-11-02 09:34:50.80154649 +0000 UTC m=+8.610090692"
Nov 02 09:34:51.669070 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:51.668998    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:51.669199 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:51.669136    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:34:51.670392 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:51.669567    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:51.670392 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:51.669670    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:34:51.789153 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:51.789114    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd" event={"ID":"cd36afed-d103-4f7a-ba99-00c841bb9537","Type":"ContainerStarted","Data":"2846692477b5608a0930f517c60592ada0649f69efd1f890a6fc8b2b0fdfeddf"}
Nov 02 09:34:52.225482 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:52.225427    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:52.225672 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:52.225538    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:52.225778 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:52.225681    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:52.225778 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:52.225746    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs podName:d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa nodeName:}" failed. No retries permitted until 2026-11-02 09:35:00.225732892 +0000 UTC m=+18.034277084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs") pod "network-metrics-daemon-p8bj8" (UID: "d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:52.225862 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:52.225797    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:52.225862 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:52.225819    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret podName:69f3517a-6eb0-4b5a-a5de-fa56577626be nodeName:}" failed. No retries permitted until 2026-11-02 09:35:00.22581278 +0000 UTC m=+18.034356971 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret") pod "global-pull-secret-syncer-8w4cb" (UID: "69f3517a-6eb0-4b5a-a5de-fa56577626be") : object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:52.427426 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:52.427378    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bjq\" (UniqueName: \"kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq\") pod \"network-check-target-2gvpb\" (UID: \"10db22d0-6805-4d79-a670-e7b96a3b9c22\") " pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:52.427615 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:52.427594    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 09:34:52.427701 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:52.427618    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 09:34:52.427701 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:52.427631    2546 projected.go:196] Error preparing data for projected volume kube-api-access-j7bjq for pod openshift-network-diagnostics/network-check-target-2gvpb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:52.427701 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:52.427690    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq podName:10db22d0-6805-4d79-a670-e7b96a3b9c22 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:00.42767285 +0000 UTC m=+18.236217041 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j7bjq" (UniqueName: "kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq") pod "network-check-target-2gvpb" (UID: "10db22d0-6805-4d79-a670-e7b96a3b9c22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:52.671419 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:52.671389    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:52.671607 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:52.671508    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:34:52.698877 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:52.698831    2546 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 09:34:52.794584 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:52.794553    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd" event={"ID":"cd36afed-d103-4f7a-ba99-00c841bb9537","Type":"ContainerStarted","Data":"9c7f30c7e92fe92e4fcc0be097b3975c1536da6d9ded029b5f783156e07a4297"}
Nov 02 09:34:52.863375 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:52.863342    2546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:34:52.864584 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:52.864562    2546 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:34:52.881932 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:52.881877    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hxwln" podStartSLOduration=4.519138429 podStartE2EDuration="9.881859499s" totalImagesPullingTime="5.36272107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:50.801844052 +0000 UTC m=+8.610388258" watchObservedRunningTime="2026-11-02 09:34:52.881859499 +0000 UTC m=+10.690403701"
Nov 02 09:34:53.643390 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:53.643056    2546 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T09:34:52.698855629Z","UUID":"68a3e01f-b06c-40c5-a700-1aac8c4910d7","Handler":null,"Name":"","Endpoint":""}
Nov 02 09:34:53.645095 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:53.644791    2546 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 09:34:53.645095 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:53.644827    2546 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 09:34:53.668655 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:53.668623    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:53.668785 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:53.668726    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:34:53.668849 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:53.668802    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:53.668881 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:53.668857    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:34:53.798737 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:53.798709    2546 generic.go:356] "Generic (PLEG): container finished" podID="18e3a7af-aafb-4491-b696-95f4858b585a" containerID="f539a73f4ce7ddf87b392003d5b6a572f8d5cc0b0f40618bdc8eba663dedffb0" exitCode=0
Nov 02 09:34:53.799089 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:53.798792    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbvj" event={"ID":"18e3a7af-aafb-4491-b696-95f4858b585a","Type":"ContainerDied","Data":"f539a73f4ce7ddf87b392003d5b6a572f8d5cc0b0f40618bdc8eba663dedffb0"}
Nov 02 09:34:53.805767 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:53.805655    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r85rg" event={"ID":"8e9ee3cd-e542-40a1-a18f-2843dcd82a69","Type":"ContainerStarted","Data":"705e24448e7faefb626e128775ccc021838b52259b292ff5e9185a34c75332b0"}
Nov 02 09:34:53.808304 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:53.808202    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vkr2t" event={"ID":"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab","Type":"ContainerStarted","Data":"11faa2580fa152c538376512c3a897362eec24e09d62df7af22f5eda9626bdde"}
Nov 02 09:34:53.808304 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:53.808266    2546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:34:53.808622 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:53.808603    2546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:34:53.855611 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:53.855558    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-r85rg" podStartSLOduration=2.739753429 podStartE2EDuration="10.855539659s" totalImagesPullingTime="8.11578623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:53.854771024 +0000 UTC m=+11.663315225" watchObservedRunningTime="2026-11-02 09:34:53.855539659 +0000 UTC m=+11.664083862"
Nov 02 09:34:53.875452 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:53.875405    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-vkr2t" podStartSLOduration=2.777511519 podStartE2EDuration="10.875387803s" totalImagesPullingTime="8.097876284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:53.874986782 +0000 UTC m=+11.683530986" watchObservedRunningTime="2026-11-02 09:34:53.875387803 +0000 UTC m=+11.683932003"
Nov 02 09:34:54.669318 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:54.669284    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:54.669477 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:54.669434    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:34:54.813209 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:54.813179    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd" event={"ID":"cd36afed-d103-4f7a-ba99-00c841bb9537","Type":"ContainerStarted","Data":"3934a81571442b6189aa3f79646e811285ae4a437b49c8e8e194a728a39c2a6d"}
Nov 02 09:34:55.668917 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:55.668888    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:55.669046 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:55.669015    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:34:55.669105 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:55.669080    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:55.669236 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:55.669197    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:34:55.817859 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:55.817783    2546 generic.go:356] "Generic (PLEG): container finished" podID="18e3a7af-aafb-4491-b696-95f4858b585a" containerID="b5e937b028cd46cc80f79dd19de80161d719a54763a0fa3190d629ae72233627" exitCode=0
Nov 02 09:34:55.818537 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:55.817875    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbvj" event={"ID":"18e3a7af-aafb-4491-b696-95f4858b585a","Type":"ContainerDied","Data":"b5e937b028cd46cc80f79dd19de80161d719a54763a0fa3190d629ae72233627"}
Nov 02 09:34:55.820239 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:55.819577    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jxnjv" event={"ID":"50bebf56-2ddc-47e0-b8f7-008d202e3af3","Type":"ContainerStarted","Data":"a17e44e9cdfc6d327708363e28131cc6f70dc8cb5f8387fa5c9356588529fb34"}
Nov 02 09:34:55.821205 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:55.821180    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r8j97" event={"ID":"5cf1aef2-0bc5-445e-880d-90215888057a","Type":"ContainerStarted","Data":"fc979316863581184d8c1174f29e9fe34502b8798b223726e51c31f0c16e0171"}
Nov 02 09:34:55.845349 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:55.845309    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd" podStartSLOduration=4.370750411 podStartE2EDuration="12.845297485s" totalImagesPullingTime="8.474547074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:54.83449637 +0000 UTC m=+12.643040571" watchObservedRunningTime="2026-11-02 09:34:55.845297485 +0000 UTC m=+13.653841736"
Nov 02 09:34:55.861527 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:55.861491    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r8j97" podStartSLOduration=4.387531029 podStartE2EDuration="12.861479366s" totalImagesPullingTime="8.473948337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:55.861312327 +0000 UTC m=+13.669856527" watchObservedRunningTime="2026-11-02 09:34:55.861479366 +0000 UTC m=+13.670023566"
Nov 02 09:34:55.880498 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:55.880459    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jxnjv" podStartSLOduration=2.19877981 podStartE2EDuration="12.880450791s" totalImagesPullingTime="10.681670981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:55.879968048 +0000 UTC m=+13.688512247" watchObservedRunningTime="2026-11-02 09:34:55.880450791 +0000 UTC m=+13.688995022"
Nov 02 09:34:56.669470 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:56.669439    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:56.669615 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:56.669570    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:34:57.669503 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:57.669281    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:57.669965 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:57.669310    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:57.669965 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:57.669580    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:34:57.669965 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:57.669677    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:34:57.828086 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:57.828052    2546 generic.go:356] "Generic (PLEG): container finished" podID="18e3a7af-aafb-4491-b696-95f4858b585a" containerID="e3c8617964cce559b85b2da5f469dbb7ebee7e00667e36c9d75c9620dd437a6e" exitCode=0
Nov 02 09:34:57.828260 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:57.828115    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbvj" event={"ID":"18e3a7af-aafb-4491-b696-95f4858b585a","Type":"ContainerDied","Data":"e3c8617964cce559b85b2da5f469dbb7ebee7e00667e36c9d75c9620dd437a6e"}
Nov 02 09:34:58.668903 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:58.668862    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:34:58.669081 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:58.669017    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:34:59.668705 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:59.668672    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:34:59.668705 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:34:59.668706    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:34:59.669197 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:59.668812    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:34:59.669197 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:34:59.668974    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:35:00.287523 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:00.287358    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:00.287646 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:00.287561    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:00.287646 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:00.287477    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:35:00.287724 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:00.287644    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs podName:d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa nodeName:}" failed. No retries permitted until 2026-11-02 09:35:16.287623769 +0000 UTC m=+34.096167965 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs") pod "network-metrics-daemon-p8bj8" (UID: "d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:35:00.287724 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:00.287691    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 09:35:00.287801 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:00.287742    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret podName:69f3517a-6eb0-4b5a-a5de-fa56577626be nodeName:}" failed. No retries permitted until 2026-11-02 09:35:16.287727734 +0000 UTC m=+34.096271935 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret") pod "global-pull-secret-syncer-8w4cb" (UID: "69f3517a-6eb0-4b5a-a5de-fa56577626be") : object "kube-system"/"original-pull-secret" not registered
Nov 02 09:35:00.489052 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:00.489024    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bjq\" (UniqueName: \"kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq\") pod \"network-check-target-2gvpb\" (UID: \"10db22d0-6805-4d79-a670-e7b96a3b9c22\") " pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:00.489160 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:00.489142    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 09:35:00.489160 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:00.489154    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 09:35:00.489267 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:00.489163    2546 projected.go:196] Error preparing data for projected volume kube-api-access-j7bjq for pod openshift-network-diagnostics/network-check-target-2gvpb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:35:00.489267 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:00.489203    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq podName:10db22d0-6805-4d79-a670-e7b96a3b9c22 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:16.489191362 +0000 UTC m=+34.297735544 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j7bjq" (UniqueName: "kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq") pod "network-check-target-2gvpb" (UID: "10db22d0-6805-4d79-a670-e7b96a3b9c22") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:35:00.671457 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:00.671433    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:00.671849 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:00.671554    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:35:00.836899 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:00.836874    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck" event={"ID":"260f9ad0-a637-4423-ac4c-626c79e6bca9","Type":"ContainerStarted","Data":"4b2d196309e8a2262f5a40d4794d154ba6b96941f308bb823dfd52cb59c40b47"}
Nov 02 09:35:00.837011 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:00.836941    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck" event={"ID":"260f9ad0-a637-4423-ac4c-626c79e6bca9","Type":"ContainerStarted","Data":"8cc0e862094d5940e46f3b8da7e60c320daf5313b51ad4fdaa613dd8bf5bd57d"}
Nov 02 09:35:00.837011 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:00.836985    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck" event={"ID":"260f9ad0-a637-4423-ac4c-626c79e6bca9","Type":"ContainerStarted","Data":"996925a7f007ac50cfa77a13bbce832ddcfffd4e506557627506b35b635fd66f"}
Nov 02 09:35:00.837110 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:00.837021    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck" event={"ID":"260f9ad0-a637-4423-ac4c-626c79e6bca9","Type":"ContainerStarted","Data":"f7b508a5f40d87c64af368af7c04792f4810729719f1e79d2ab5072d5a422167"}
Nov 02 09:35:00.837110 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:00.837047    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck" event={"ID":"260f9ad0-a637-4423-ac4c-626c79e6bca9","Type":"ContainerStarted","Data":"04a9d6154fce8f7efe690e4991da50dc7d354de1a0fd541fcc95e6a86ec93880"}
Nov 02 09:35:00.837110 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:00.837073    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck" event={"ID":"260f9ad0-a637-4423-ac4c-626c79e6bca9","Type":"ContainerStarted","Data":"de720e93e8df5e0e97199dac4a9d2ae1bce24a92276e5047a7581a9e525388df"}
Nov 02 09:35:00.839431 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:00.839410    2546 generic.go:356] "Generic (PLEG): container finished" podID="18e3a7af-aafb-4491-b696-95f4858b585a" containerID="4b88618acde430f4002965370f2a88d17bc57f166a3f24dd3e20a4f5678e67c7" exitCode=0
Nov 02 09:35:00.839498 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:00.839447    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbvj" event={"ID":"18e3a7af-aafb-4491-b696-95f4858b585a","Type":"ContainerDied","Data":"4b88618acde430f4002965370f2a88d17bc57f166a3f24dd3e20a4f5678e67c7"}
Nov 02 09:35:01.668944 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:01.668898    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:01.668944 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:01.668937    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:01.669116 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:01.669013    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:35:01.669174 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:01.669119    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:35:01.844422 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:01.844385    2546 generic.go:356] "Generic (PLEG): container finished" podID="18e3a7af-aafb-4491-b696-95f4858b585a" containerID="e348a68294de81a65c9e7b5ba2b75510d4a6d84c8c84018d2fbd7c5c0a889bb2" exitCode=0
Nov 02 09:35:01.844893 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:01.844441    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbvj" event={"ID":"18e3a7af-aafb-4491-b696-95f4858b585a","Type":"ContainerDied","Data":"e348a68294de81a65c9e7b5ba2b75510d4a6d84c8c84018d2fbd7c5c0a889bb2"}
Nov 02 09:35:02.669426 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:02.669385    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:02.669590 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:02.669490    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:35:02.849469 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:02.849434    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck" event={"ID":"260f9ad0-a637-4423-ac4c-626c79e6bca9","Type":"ContainerStarted","Data":"eb1098339588e187d93043339f437369f79a9f5412b758ee1e61693212c3a498"}
Nov 02 09:35:02.852602 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:02.852579    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8pbvj" event={"ID":"18e3a7af-aafb-4491-b696-95f4858b585a","Type":"ContainerStarted","Data":"5b23b7578fd3d4be1c38a6d6a2b9a504225c359eccd638fa1b708cc0e115a1b0"}
Nov 02 09:35:02.878718 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:02.878677    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8pbvj" podStartSLOduration=8.287563829 podStartE2EDuration="19.87866666s" totalImagesPullingTime="11.591102831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:02.877032139 +0000 UTC m=+20.685576338" watchObservedRunningTime="2026-11-02 09:35:02.87866666 +0000 UTC m=+20.687210861"
Nov 02 09:35:03.669019 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:03.668986    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:03.669195 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:03.669110    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:35:03.669195 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:03.669157    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:03.669365 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:03.669250    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:35:04.672008 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:04.671983    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:04.672526 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:04.672110    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:35:04.860627 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:04.860369    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck" event={"ID":"260f9ad0-a637-4423-ac4c-626c79e6bca9","Type":"ContainerStarted","Data":"6969717e65886e20cfe677cf204929e26706d9c4830292aa891601d7bdc44d46"}
Nov 02 09:35:04.860627 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:04.860614    2546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:04.876464 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:04.876439    2546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:04.906559 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:04.906523    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck" podStartSLOduration=6.740464544 podStartE2EDuration="21.906512615s" totalImagesPullingTime="15.166048071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:04.891624689 +0000 UTC m=+22.700168889" watchObservedRunningTime="2026-11-02 09:35:04.906512615 +0000 UTC m=+22.715056838"
Nov 02 09:35:05.101440 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:05.101412    2546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:05.668752 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:05.668725    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:05.668752 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:05.668756    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:05.669000 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:05.668825    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:35:05.669000 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:05.668874    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:35:05.863004 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:05.862966    2546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:05.894251 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:05.884540    2546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:06.427714 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:06.427630    2546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2gvpb"]
Nov 02 09:35:06.427855 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:06.427758    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:06.427921 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:06.427894    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:35:06.428272 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:06.428249    2546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8w4cb"]
Nov 02 09:35:06.428391 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:06.428334    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:06.428451 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:06.428422    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:35:06.429633 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:06.429610    2546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8bj8"]
Nov 02 09:35:06.429723 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:06.429712    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:06.429804 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:06.429790    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:35:07.669057 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:07.669019    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:07.669057 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:07.669055    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:07.669558 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:07.669123    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:35:07.669558 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:07.669141    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:07.669558 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:07.669251    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:35:07.669558 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:07.669290    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:35:09.668743 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:09.668659    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:09.668743 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:09.668689    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:09.669241 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:09.668771    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:35:09.669241 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:09.668816    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:09.669241 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:09.668895    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:35:09.669241 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:09.668956    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:35:11.669497 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:11.669311    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:11.669876 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:11.669332    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:11.669876 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:11.669581    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8bj8" podUID="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa"
Nov 02 09:35:11.669876 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:11.669363    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:11.669876 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:11.669644    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w4cb" podUID="69f3517a-6eb0-4b5a-a5de-fa56577626be"
Nov 02 09:35:11.669876 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:11.669714    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gvpb" podUID="10db22d0-6805-4d79-a670-e7b96a3b9c22"
Nov 02 09:35:12.470991 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.470963    2546 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeReady"
Nov 02 09:35:12.471139 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.471123    2546 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 09:35:12.517756 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.517727    2546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sjj6v"]
Nov 02 09:35:12.520453 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.520422    2546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l7v8r"]
Nov 02 09:35:12.520581 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.520529    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:12.523299 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.523280    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l7v8r"
Nov 02 09:35:12.529328 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.529306    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 09:35:12.532429 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.532384    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 09:35:12.532575 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.532507    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 09:35:12.532575 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.532547    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gztjj\""
Nov 02 09:35:12.532575 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.532585    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zwj6d\""
Nov 02 09:35:12.534833 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.534815    2546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sjj6v"]
Nov 02 09:35:12.535520 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.535503    2546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l7v8r"]
Nov 02 09:35:12.630252 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.630155    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 09:35:12.630252 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.630191    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 09:35:12.678580 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.678560    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60edc0c2-d978-40b5-b155-152019a3bdf5-config-volume\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:12.678911 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.678591    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbngm\" (UniqueName: \"kubernetes.io/projected/b0a20d22-7ea2-4cb8-a46f-d48043889d6f-kube-api-access-lbngm\") pod \"ingress-canary-l7v8r\" (UID: \"b0a20d22-7ea2-4cb8-a46f-d48043889d6f\") " pod="openshift-ingress-canary/ingress-canary-l7v8r"
Nov 02 09:35:12.678911 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.678627    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60edc0c2-d978-40b5-b155-152019a3bdf5-tmp-dir\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:12.678911 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.678758    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwnd7\" (UniqueName: \"kubernetes.io/projected/60edc0c2-d978-40b5-b155-152019a3bdf5-kube-api-access-wwnd7\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:12.678911 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.678798    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0a20d22-7ea2-4cb8-a46f-d48043889d6f-cert\") pod \"ingress-canary-l7v8r\" (UID: \"b0a20d22-7ea2-4cb8-a46f-d48043889d6f\") " pod="openshift-ingress-canary/ingress-canary-l7v8r"
Nov 02 09:35:12.678911 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.678815    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:12.779242 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.779202    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60edc0c2-d978-40b5-b155-152019a3bdf5-tmp-dir\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:12.779357 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.779260    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwnd7\" (UniqueName: \"kubernetes.io/projected/60edc0c2-d978-40b5-b155-152019a3bdf5-kube-api-access-wwnd7\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:12.779357 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.779288    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0a20d22-7ea2-4cb8-a46f-d48043889d6f-cert\") pod \"ingress-canary-l7v8r\" (UID: \"b0a20d22-7ea2-4cb8-a46f-d48043889d6f\") " pod="openshift-ingress-canary/ingress-canary-l7v8r"
Nov 02 09:35:12.779357 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.779303    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:12.779357 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.779337    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60edc0c2-d978-40b5-b155-152019a3bdf5-config-volume\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:12.779597 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.779367    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbngm\" (UniqueName: \"kubernetes.io/projected/b0a20d22-7ea2-4cb8-a46f-d48043889d6f-kube-api-access-lbngm\") pod \"ingress-canary-l7v8r\" (UID: \"b0a20d22-7ea2-4cb8-a46f-d48043889d6f\") " pod="openshift-ingress-canary/ingress-canary-l7v8r"
Nov 02 09:35:12.779597 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:12.779429    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:12.779597 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:12.779497    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls podName:60edc0c2-d978-40b5-b155-152019a3bdf5 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:13.279478708 +0000 UTC m=+31.088022889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls") pod "dns-default-sjj6v" (UID: "60edc0c2-d978-40b5-b155-152019a3bdf5") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:12.779771 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.779749    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60edc0c2-d978-40b5-b155-152019a3bdf5-tmp-dir\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:12.780014 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.779996    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60edc0c2-d978-40b5-b155-152019a3bdf5-config-volume\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:12.783366 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.783345    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0a20d22-7ea2-4cb8-a46f-d48043889d6f-cert\") pod \"ingress-canary-l7v8r\" (UID: \"b0a20d22-7ea2-4cb8-a46f-d48043889d6f\") " pod="openshift-ingress-canary/ingress-canary-l7v8r"
Nov 02 09:35:12.791421 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.791401    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwnd7\" (UniqueName: \"kubernetes.io/projected/60edc0c2-d978-40b5-b155-152019a3bdf5-kube-api-access-wwnd7\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:12.792136 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.792121    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbngm\" (UniqueName: \"kubernetes.io/projected/b0a20d22-7ea2-4cb8-a46f-d48043889d6f-kube-api-access-lbngm\") pod \"ingress-canary-l7v8r\" (UID: \"b0a20d22-7ea2-4cb8-a46f-d48043889d6f\") " pod="openshift-ingress-canary/ingress-canary-l7v8r"
Nov 02 09:35:12.840123 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:12.840090    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l7v8r"
Nov 02 09:35:13.124040 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:13.124012    2546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l7v8r"]
Nov 02 09:35:13.127431 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:35:13.127409    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a20d22_7ea2_4cb8_a46f_d48043889d6f.slice/crio-d912ba6609f62041b568e2fd0d24f5c414b04c4d859a3463397de5032b0b3d97 WatchSource:0}: Error finding container d912ba6609f62041b568e2fd0d24f5c414b04c4d859a3463397de5032b0b3d97: Status 404 returned error can't find the container with id d912ba6609f62041b568e2fd0d24f5c414b04c4d859a3463397de5032b0b3d97
Nov 02 09:35:13.283230 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:13.283164    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:13.283308 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:13.283297    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:13.283366 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:13.283357    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls podName:60edc0c2-d978-40b5-b155-152019a3bdf5 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:14.283339824 +0000 UTC m=+32.091884004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls") pod "dns-default-sjj6v" (UID: "60edc0c2-d978-40b5-b155-152019a3bdf5") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:13.668880 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:13.668850    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:13.668880 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:13.668881    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:13.669023 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:13.668881    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:13.672883 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:13.672863    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 09:35:13.672883 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:13.672876    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 09:35:13.672883 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:13.672882    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 09:35:13.674251 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:13.674207    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 09:35:13.674251 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:13.674207    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9d4cl\""
Nov 02 09:35:13.674537 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:13.674524    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbhf4\""
Nov 02 09:35:13.674621 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:13.674603    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sbddj\""
Nov 02 09:35:13.880270 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:13.880233    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l7v8r" event={"ID":"b0a20d22-7ea2-4cb8-a46f-d48043889d6f","Type":"ContainerStarted","Data":"d912ba6609f62041b568e2fd0d24f5c414b04c4d859a3463397de5032b0b3d97"}
Nov 02 09:35:14.291475 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:14.291439    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:14.291612 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:14.291565    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:14.291663 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:14.291619    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls podName:60edc0c2-d978-40b5-b155-152019a3bdf5 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:16.291605989 +0000 UTC m=+34.100150167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls") pod "dns-default-sjj6v" (UID: "60edc0c2-d978-40b5-b155-152019a3bdf5") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:16.307868 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.307825    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:16.308340 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.307884    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:16.308340 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.307916    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:16.308340 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:16.307971    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 09:35:16.308340 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:16.308027    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:16.308340 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:16.308044    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs podName:d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa nodeName:}" failed. No retries permitted until 2026-11-02 09:35:48.308023101 +0000 UTC m=+66.116567279 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs") pod "network-metrics-daemon-p8bj8" (UID: "d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa") : secret "metrics-daemon-secret" not found
Nov 02 09:35:16.308340 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:16.308081    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls podName:60edc0c2-d978-40b5-b155-152019a3bdf5 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:20.308063678 +0000 UTC m=+38.116607880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls") pod "dns-default-sjj6v" (UID: "60edc0c2-d978-40b5-b155-152019a3bdf5") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:16.310602 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.310577    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:16.427293 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.427206    2546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-smlj2"]
Nov 02 09:35:16.436388 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.436361    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:16.490154 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.490132    2546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-pgbtq-2c6rz"]
Nov 02 09:35:16.490308 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.490283    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.503031 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.503013    2546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-smlj2"]
Nov 02 09:35:16.503139 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.503049    2546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pgbtq-2c6rz"]
Nov 02 09:35:16.503199 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.503138    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.509381 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.509353    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bjq\" (UniqueName: \"kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq\") pod \"network-check-target-2gvpb\" (UID: \"10db22d0-6805-4d79-a670-e7b96a3b9c22\") " pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:16.512383 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.512363    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7bjq\" (UniqueName: \"kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq\") pod \"network-check-target-2gvpb\" (UID: \"10db22d0-6805-4d79-a670-e7b96a3b9c22\") " pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:16.555112 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.555076    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 09:35:16.555296 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.555282    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 09:35:16.555419 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.555402    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 09:35:16.555487 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.555475    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 09:35:16.558484 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.558456    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 09:35:16.561261 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.561236    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fzpc9\""
Nov 02 09:35:16.561412 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.561393    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 09:35:16.561480 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.561453    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xrsh4\""
Nov 02 09:35:16.609891 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.609862    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c108bc48-9b97-46d1-a353-04a6dbc9ea83-crio-socket\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.609998 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.609916    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c108bc48-9b97-46d1-a353-04a6dbc9ea83-data-volume\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.609998 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.609946    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-service-ca-bundle\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.609998 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.609966    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.610127 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.610068    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lptr9\" (UniqueName: \"kubernetes.io/projected/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-api-access-lptr9\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.610127 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.610086    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/86739438-2924-4642-9d34-ed545e1739ba-archives-path\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.610127 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.610102    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsmrc\" (UniqueName: \"kubernetes.io/projected/86739438-2924-4642-9d34-ed545e1739ba-kube-api-access-lsmrc\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.610127 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.610123    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-trusted-ca-bundle\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.610287 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.610244    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.710650 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.710582    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-trusted-ca-bundle\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.710650 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.710641    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.710825 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.710691    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c108bc48-9b97-46d1-a353-04a6dbc9ea83-crio-socket\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.710825 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.710731    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c108bc48-9b97-46d1-a353-04a6dbc9ea83-data-volume\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.710825 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.710754    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-service-ca-bundle\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.710825 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:16.710775    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 09:35:16.711022 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:16.710848    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm podName:c108bc48-9b97-46d1-a353-04a6dbc9ea83 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:17.210828406 +0000 UTC m=+35.019372607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-smlj2" (UID: "c108bc48-9b97-46d1-a353-04a6dbc9ea83") : configmap "kube-rbac-proxy" not found
Nov 02 09:35:16.711022 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:16.710867    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:16.711022 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.710902    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c108bc48-9b97-46d1-a353-04a6dbc9ea83-crio-socket\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.711022 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:16.710929    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls podName:c108bc48-9b97-46d1-a353-04a6dbc9ea83 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:17.210912909 +0000 UTC m=+35.019457104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smlj2" (UID: "c108bc48-9b97-46d1-a353-04a6dbc9ea83") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:16.711022 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.710780    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.711022 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.710978    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lptr9\" (UniqueName: \"kubernetes.io/projected/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-api-access-lptr9\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.711022 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.711009    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/86739438-2924-4642-9d34-ed545e1739ba-archives-path\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.711466 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.711036    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsmrc\" (UniqueName: \"kubernetes.io/projected/86739438-2924-4642-9d34-ed545e1739ba-kube-api-access-lsmrc\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.711466 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.711145    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c108bc48-9b97-46d1-a353-04a6dbc9ea83-data-volume\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.711466 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.711456    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-trusted-ca-bundle\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.711585 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.711477    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-service-ca-bundle\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.711585 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.711483    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/86739438-2924-4642-9d34-ed545e1739ba-archives-path\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.718570 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.718549    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:16.729717 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.729691    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsmrc\" (UniqueName: \"kubernetes.io/projected/86739438-2924-4642-9d34-ed545e1739ba-kube-api-access-lsmrc\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.731034 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.731010    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lptr9\" (UniqueName: \"kubernetes.io/projected/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-api-access-lptr9\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:16.773418 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.773393    2546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8w4cb"]
Nov 02 09:35:16.776331 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:35:16.776299    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f3517a_6eb0_4b5a_a5de_fa56577626be.slice/crio-6636d3d7808706e48bad250142326b9f47a56d1b998aa688a79e1bf2b97a77d1 WatchSource:0}: Error finding container 6636d3d7808706e48bad250142326b9f47a56d1b998aa688a79e1bf2b97a77d1: Status 404 returned error can't find the container with id 6636d3d7808706e48bad250142326b9f47a56d1b998aa688a79e1bf2b97a77d1
Nov 02 09:35:16.832570 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.832535    2546 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz"]
Nov 02 09:35:16.852052 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.852029    2546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz"]
Nov 02 09:35:16.852153 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.852109    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz"
Nov 02 09:35:16.875244 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.875206    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:16.886991 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.886970    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8w4cb" event={"ID":"69f3517a-6eb0-4b5a-a5de-fa56577626be","Type":"ContainerStarted","Data":"6636d3d7808706e48bad250142326b9f47a56d1b998aa688a79e1bf2b97a77d1"}
Nov 02 09:35:16.901657 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.901640    2546 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6hxzv\""
Nov 02 09:35:16.915404 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.915383    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:16.915574 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:16.915557    2546 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 09:35:17.013860 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:17.013801    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl5qh\" (UniqueName: \"kubernetes.io/projected/1ee8e64e-98c3-442e-9885-f45afd29a812-kube-api-access-sl5qh\") pod \"migrator-66df5d496b-ng2mz\" (UID: \"1ee8e64e-98c3-442e-9885-f45afd29a812\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz"
Nov 02 09:35:17.017654 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:17.017631    2546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2gvpb"]
Nov 02 09:35:17.114146 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:17.114117    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5qh\" (UniqueName: \"kubernetes.io/projected/1ee8e64e-98c3-442e-9885-f45afd29a812-kube-api-access-sl5qh\") pod \"migrator-66df5d496b-ng2mz\" (UID: \"1ee8e64e-98c3-442e-9885-f45afd29a812\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz"
Nov 02 09:35:17.126138 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:17.126116    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5qh\" (UniqueName: \"kubernetes.io/projected/1ee8e64e-98c3-442e-9885-f45afd29a812-kube-api-access-sl5qh\") pod \"migrator-66df5d496b-ng2mz\" (UID: \"1ee8e64e-98c3-442e-9885-f45afd29a812\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz"
Nov 02 09:35:17.172101 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:17.172074    2546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pgbtq-2c6rz"]
Nov 02 09:35:17.172293 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:35:17.172271    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10db22d0_6805_4d79_a670_e7b96a3b9c22.slice/crio-2c827b8260d7b755e6d4866b18d443871095b032fc08bedfd6348b4fd0d77e1c WatchSource:0}: Error finding container 2c827b8260d7b755e6d4866b18d443871095b032fc08bedfd6348b4fd0d77e1c: Status 404 returned error can't find the container with id 2c827b8260d7b755e6d4866b18d443871095b032fc08bedfd6348b4fd0d77e1c
Nov 02 09:35:17.174642 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:17.174624    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz"
Nov 02 09:35:17.174920 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:35:17.174902    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86739438_2924_4642_9d34_ed545e1739ba.slice/crio-1cc09061c97b92a0cbebcd9d3aedb18b9004d60918452703e9c14d89714ae540 WatchSource:0}: Error finding container 1cc09061c97b92a0cbebcd9d3aedb18b9004d60918452703e9c14d89714ae540: Status 404 returned error can't find the container with id 1cc09061c97b92a0cbebcd9d3aedb18b9004d60918452703e9c14d89714ae540
Nov 02 09:35:17.215018 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:17.214984    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:17.215127 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:17.215078    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:17.215127 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:17.215099    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 09:35:17.215228 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:17.215167    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm podName:c108bc48-9b97-46d1-a353-04a6dbc9ea83 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:18.21514634 +0000 UTC m=+36.023690535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-smlj2" (UID: "c108bc48-9b97-46d1-a353-04a6dbc9ea83") : configmap "kube-rbac-proxy" not found
Nov 02 09:35:17.215228 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:17.215181    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:17.215339 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:17.215243    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls podName:c108bc48-9b97-46d1-a353-04a6dbc9ea83 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:18.21520713 +0000 UTC m=+36.023751328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smlj2" (UID: "c108bc48-9b97-46d1-a353-04a6dbc9ea83") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:17.462859 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:17.462828    2546 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz"]
Nov 02 09:35:17.466733 ip-10-0-128-217 kubenswrapper[2546]: W1102 09:35:17.466696    2546 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee8e64e_98c3_442e_9885_f45afd29a812.slice/crio-d13ce16fb67c2d2ded6fb06c09753f0bcbde3a067663ef5d3641dbc97fc69783 WatchSource:0}: Error finding container d13ce16fb67c2d2ded6fb06c09753f0bcbde3a067663ef5d3641dbc97fc69783: Status 404 returned error can't find the container with id d13ce16fb67c2d2ded6fb06c09753f0bcbde3a067663ef5d3641dbc97fc69783
Nov 02 09:35:17.893780 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:17.893738    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz" event={"ID":"1ee8e64e-98c3-442e-9885-f45afd29a812","Type":"ContainerStarted","Data":"d13ce16fb67c2d2ded6fb06c09753f0bcbde3a067663ef5d3641dbc97fc69783"}
Nov 02 09:35:17.896339 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:17.896277    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2gvpb" event={"ID":"10db22d0-6805-4d79-a670-e7b96a3b9c22","Type":"ContainerStarted","Data":"2c827b8260d7b755e6d4866b18d443871095b032fc08bedfd6348b4fd0d77e1c"}
Nov 02 09:35:17.898736 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:17.898664    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pgbtq-2c6rz" event={"ID":"86739438-2924-4642-9d34-ed545e1739ba","Type":"ContainerStarted","Data":"1cc09061c97b92a0cbebcd9d3aedb18b9004d60918452703e9c14d89714ae540"}
Nov 02 09:35:17.901269 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:17.901234    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l7v8r" event={"ID":"b0a20d22-7ea2-4cb8-a46f-d48043889d6f","Type":"ContainerStarted","Data":"d2dda41af92968fadc1911d25550a2c97d60ab9aee95cfd40f20d14b54afb584"}
Nov 02 09:35:18.224903 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:18.224819    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:18.225052 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:18.224914    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:18.225052 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:18.224936    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 09:35:18.225052 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:18.225014    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm podName:c108bc48-9b97-46d1-a353-04a6dbc9ea83 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:20.22499229 +0000 UTC m=+38.033536474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-smlj2" (UID: "c108bc48-9b97-46d1-a353-04a6dbc9ea83") : configmap "kube-rbac-proxy" not found
Nov 02 09:35:18.225052 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:18.225025    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:18.225248 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:18.225072    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls podName:c108bc48-9b97-46d1-a353-04a6dbc9ea83 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:20.225059175 +0000 UTC m=+38.033603367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smlj2" (UID: "c108bc48-9b97-46d1-a353-04a6dbc9ea83") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:19.909494 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:19.909453    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz" event={"ID":"1ee8e64e-98c3-442e-9885-f45afd29a812","Type":"ContainerStarted","Data":"57b46b5caf4b8a16bc4d1cdffe044fa18f589d2af6eb67c9f7016ff82d978675"}
Nov 02 09:35:19.927300 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:19.927245    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l7v8r" podStartSLOduration=3.849671566 podStartE2EDuration="7.927210369s" totalImagesPullingTime="4.077538803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:17.921835268 +0000 UTC m=+35.730379469" watchObservedRunningTime="2026-11-02 09:35:19.927210369 +0000 UTC m=+37.735754569"
Nov 02 09:35:19.927511 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:19.927356    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz" podStartSLOduration=2.020373917 podStartE2EDuration="3.92734874s" totalImagesPullingTime="1.906974823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:19.926955484 +0000 UTC m=+37.735499687" watchObservedRunningTime="2026-11-02 09:35:19.92734874 +0000 UTC m=+37.735892941"
Nov 02 09:35:20.240700 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:20.240661    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:20.240856 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:20.240723    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:20.240856 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:20.240829    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:20.240966 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:20.240872    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls podName:c108bc48-9b97-46d1-a353-04a6dbc9ea83 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.240859018 +0000 UTC m=+42.049403199 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smlj2" (UID: "c108bc48-9b97-46d1-a353-04a6dbc9ea83") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:20.241508 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:20.241486    2546 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:20.341422 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:20.341384    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:20.341650 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:20.341555    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:20.341650 ip-10-0-128-217 kubenswrapper[2546]: E1102 09:35:20.341629    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls podName:60edc0c2-d978-40b5-b155-152019a3bdf5 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:28.341608461 +0000 UTC m=+46.150152646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls") pod "dns-default-sjj6v" (UID: "60edc0c2-d978-40b5-b155-152019a3bdf5") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:20.914428 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:20.914346    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pgbtq-2c6rz" event={"ID":"86739438-2924-4642-9d34-ed545e1739ba","Type":"ContainerStarted","Data":"5d6fb61fed161d8ddc64effbc014568a83458f25561283646424eeeff4d8c0f6"}
Nov 02 09:35:21.394372 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:21.394338    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h48j7_ef320c69-127d-437a-87ed-cfcd9db66325/konnectivity-agent/0.log" limit=1024
Nov 02 09:35:21.919304 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:21.919254    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2gvpb" event={"ID":"10db22d0-6805-4d79-a670-e7b96a3b9c22","Type":"ContainerStarted","Data":"2f30cf4a5c7cba8cf1568d696a02e5201bc210f1182ec76e00def4fab3c761ea"}
Nov 02 09:35:21.937050 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:21.936996    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2gvpb" podStartSLOduration=34.855015902 podStartE2EDuration="38.936979712s" totalImagesPullingTime="4.08196381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:21.936801514 +0000 UTC m=+39.745345718" watchObservedRunningTime="2026-11-02 09:35:21.936979712 +0000 UTC m=+39.745523907"
Nov 02 09:35:21.937394 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:21.937364    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-pgbtq-2c6rz" podStartSLOduration=3.018512455 podStartE2EDuration="5.937357553s" totalImagesPullingTime="2.918845098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:20.967558385 +0000 UTC m=+38.776102579" watchObservedRunningTime="2026-11-02 09:35:21.937357553 +0000 UTC m=+39.745901754"
Nov 02 09:35:22.263928 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:22.263835    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h48j7_ef320c69-127d-437a-87ed-cfcd9db66325/konnectivity-agent/0.log" limit=1024
Nov 02 09:35:22.922991 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:22.922953    2546 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8w4cb" event={"ID":"69f3517a-6eb0-4b5a-a5de-fa56577626be","Type":"ContainerStarted","Data":"ed5f89d36e2c6e7fd268907b9e04ae73bb421e3fe660b960d838f03c866cbd93"}
Nov 02 09:35:22.923468 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:22.923094    2546 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:22.940466 ip-10-0-128-217 kubenswrapper[2546]: I1102 09:35:22.940412    2546 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8w4cb" podStartSLOduration=33.998344753 podStartE2EDuration="39.940388029s" totalImagesPullingTime="5.942043276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:22.939691353 +0000 UTC m=+40.748235553" watchObservedRunningTime="2026-11-02 09:35:22.940388029 +0000 UTC m=+40.748932230"
Nov 02 09:35:23.008266 ip-10-0-128-217 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 09:35:23.021601 ip-10-0-128-217 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 09:35:23.021768 ip-10-0-128-217 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 09:35:23.021813 ip-10-0-128-217 systemd[1]: kubelet.service: Consumed 2.343s CPU time.
Nov 02 09:35:23.031668 ip-10-0-128-217 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 09:35:23.124104 ip-10-0-128-217 kubenswrapper[4449]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 09:35:23.124104 ip-10-0-128-217 kubenswrapper[4449]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 09:35:23.124104 ip-10-0-128-217 kubenswrapper[4449]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 09:35:23.124104 ip-10-0-128-217 kubenswrapper[4449]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 09:35:23.126139 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126064    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:23.126139 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126139    4449 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126145    4449 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126150    4449 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126154    4449 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126160    4449 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126163    4449 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126166    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126168    4449 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126171    4449 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126174    4449 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126177    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126179    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126182    4449 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126185    4449 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126187    4449 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126190    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126192    4449 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126195    4449 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126197    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:23.126211 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126200    4449 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126203    4449 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126206    4449 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126208    4449 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126211    4449 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126231    4449 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126234    4449 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126237    4449 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126240    4449 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126243    4449 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126246    4449 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126249    4449 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126252    4449 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126255    4449 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126257    4449 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126260    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126262    4449 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126265    4449 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126268    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126272    4449 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:23.126701 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126275    4449 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126277    4449 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126280    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126282    4449 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126285    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126288    4449 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126290    4449 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126293    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126296    4449 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126298    4449 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126301    4449 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126304    4449 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126307    4449 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126309    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126313    4449 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126315    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126318    4449 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126321    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126323    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:23.127197 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126326    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126329    4449 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126332    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126335    4449 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126338    4449 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126340    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126343    4449 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126346    4449 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126349    4449 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126351    4449 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126354    4449 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126357    4449 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126359    4449 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126362    4449 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126364    4449 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126367    4449 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126369    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126372    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126375    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:23.127687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126377    4449 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126380    4449 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126383    4449 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126386    4449 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126389    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126392    4449 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126395    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126397    4449 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126400    4449 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126403    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126405    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126408    4449 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126411    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126413    4449 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126416    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126418    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126421    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126423    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126426    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126428    4449 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:23.128154 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126431    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126433    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126438    4449 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126442    4449 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126445    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126449    4449 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126451    4449 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.126454    4449 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127202    4449 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127208    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127226    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127229    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127232    4449 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127235    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127238    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127241    4449 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127243    4449 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127246    4449 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127249    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:23.128664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127251    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127254    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127256    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127259    4449 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127261    4449 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127264    4449 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127267    4449 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127269    4449 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127272    4449 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127274    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127277    4449 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127281    4449 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127285    4449 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127288    4449 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127291    4449 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127294    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127296    4449 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127299    4449 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127301    4449 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:23.129143 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127305    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127308    4449 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127310    4449 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127313    4449 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127315    4449 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127318    4449 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127321    4449 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127323    4449 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127326    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127328    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127331    4449 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127334    4449 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127336    4449 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127339    4449 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127341    4449 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127344    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127347    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127349    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127352    4449 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127356    4449 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:23.129628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127359    4449 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127363    4449 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127367    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127370    4449 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127373    4449 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127375    4449 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127378    4449 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127380    4449 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127383    4449 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127386    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127392    4449 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127395    4449 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127397    4449 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127400    4449 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127403    4449 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127405    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127408    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127410    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127413    4449 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:23.130167 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127415    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127418    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127420    4449 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127423    4449 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127425    4449 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127428    4449 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127430    4449 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127433    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127435    4449 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127438    4449 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127440    4449 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127443    4449 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127445    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127448    4449 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127450    4449 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127453    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127455    4449 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127458    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127460    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127462    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:23.130648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127465    4449 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127467    4449 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127470    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127474    4449 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127477    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127479    4449 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127482    4449 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127485    4449 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127487    4449 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127490    4449 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127493    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127495    4449 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127498    4449 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127500    4449 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127503    4449 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127505    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.127508    4449 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128031    4449 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128037    4449 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:23.131142 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128039    4449 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128042    4449 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128045    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128047    4449 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128050    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128053    4449 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128072    4449 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128075    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128078    4449 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128081    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128083    4449 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128086    4449 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128088    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128100    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128102    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128105    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128110    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128112    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128115    4449 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:23.131628 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128118    4449 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128126    4449 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128129    4449 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128131    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128134    4449 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128137    4449 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128139    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128142    4449 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128144    4449 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128147    4449 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128149    4449 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128151    4449 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128154    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128157    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128160    4449 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128162    4449 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128165    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128168    4449 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128170    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:23.132083 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128173    4449 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128175    4449 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128178    4449 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128180    4449 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128183    4449 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128185    4449 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128187    4449 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128190    4449 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128192    4449 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128195    4449 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128200    4449 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128202    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128205    4449 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128207    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128210    4449 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128234    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128237    4449 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128240    4449 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128242    4449 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128245    4449 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:23.132594 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128247    4449 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128250    4449 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128252    4449 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128255    4449 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128257    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128260    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128262    4449 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128265    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128267    4449 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128272    4449 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128274    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128278    4449 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128282    4449 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128284    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128287    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128290    4449 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128292    4449 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128295    4449 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128299    4449 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:23.133316 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128302    4449 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128305    4449 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128307    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128311    4449 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128313    4449 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128316    4449 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128318    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128321    4449 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128323    4449 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128333    4449 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128336    4449 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128339    4449 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128342    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128344    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128346    4449 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128349    4449 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128351    4449 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128354    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128356    4449 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128359    4449 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128361    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:23.133976 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128364    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128367    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128370    4449 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128373    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128375    4449 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.128377    4449 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128473    4449 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128480    4449 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128495    4449 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128502    4449 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128508    4449 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128514    4449 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128519    4449 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128523    4449 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128528    4449 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128531    4449 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128534    4449 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128538    4449 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128541    4449 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128544    4449 flags.go:64] FLAG: --cgroup-root=""
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128547    4449 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128556    4449 flags.go:64] FLAG: --client-ca-file=""
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128559    4449 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 09:35:23.134511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128562    4449 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128569    4449 flags.go:64] FLAG: --cluster-domain=""
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128572    4449 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128575    4449 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128578    4449 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128581    4449 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128585    4449 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128588    4449 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128591    4449 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128594    4449 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128598    4449 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128601    4449 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128604    4449 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128607    4449 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128610    4449 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128615    4449 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128618    4449 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128621    4449 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128623    4449 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128626    4449 flags.go:64] FLAG: --enable-server="true"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128629    4449 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128636    4449 flags.go:64] FLAG: --event-burst="100"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128639    4449 flags.go:64] FLAG: --event-qps="50"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128644    4449 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 09:35:23.135067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128647    4449 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128650    4449 flags.go:64] FLAG: --eviction-hard=""
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128654    4449 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128657    4449 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128660    4449 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128663    4449 flags.go:64] FLAG: --eviction-soft=""
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128666    4449 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128675    4449 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128678    4449 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128681    4449 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128684    4449 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128687    4449 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128690    4449 flags.go:64] FLAG: --feature-gates=""
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128694    4449 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128697    4449 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128700    4449 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128703    4449 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128706    4449 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128712    4449 flags.go:64] FLAG: --help="false"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128715    4449 flags.go:64] FLAG: --hostname-override="ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128719    4449 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128722    4449 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128725    4449 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 09:35:23.135712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128728    4449 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128731    4449 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128734    4449 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128737    4449 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128740    4449 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128743    4449 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128746    4449 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128749    4449 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128753    4449 flags.go:64] FLAG: --kube-reserved=""
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128756    4449 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128758    4449 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128761    4449 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128764    4449 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128767    4449 flags.go:64] FLAG: --lock-file=""
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128770    4449 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128773    4449 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128776    4449 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128787    4449 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128790    4449 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128793    4449 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128796    4449 flags.go:64] FLAG: --logging-format="text"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128798    4449 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128802    4449 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128804    4449 flags.go:64] FLAG: --manifest-url=""
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128807    4449 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 09:35:23.136382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128811    4449 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128814    4449 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128821    4449 flags.go:64] FLAG: --max-pods="110"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128824    4449 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128827    4449 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128830    4449 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128833    4449 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128836    4449 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128839    4449 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128842    4449 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128848    4449 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128852    4449 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128855    4449 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128858    4449 flags.go:64] FLAG: --pod-cidr=""
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128861    4449 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128864    4449 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128867    4449 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128870    4449 flags.go:64] FLAG: --port="10250"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128873    4449 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128875    4449 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-083212a889d37c733"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128879    4449 flags.go:64] FLAG: --qos-reserved=""
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128881    4449 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128885    4449 flags.go:64] FLAG: --register-node="true"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128887    4449 flags.go:64] FLAG: --register-with-taints=""
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128891    4449 flags.go:64] FLAG: --registry-burst="10"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128899    4449 flags.go:64] FLAG: --registry-qps="5"
Nov 02 09:35:23.136984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128902    4449 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128905    4449 flags.go:64] FLAG: --reserved-memory=""
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128909    4449 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128912    4449 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128915    4449 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128918    4449 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128921    4449 flags.go:64] FLAG: --runonce="false"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128924    4449 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128928    4449 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128931    4449 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128934    4449 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128937    4449 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128940    4449 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128943    4449 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128946    4449 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128949    4449 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128952    4449 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128955    4449 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128958    4449 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128961    4449 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128964    4449 flags.go:64] FLAG: --system-cgroups=""
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128966    4449 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128972    4449 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128974    4449 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128977    4449 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 09:35:23.137658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128984    4449 flags.go:64] FLAG: --tls-min-version=""
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128987    4449 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128990    4449 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128993    4449 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128996    4449 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.128999    4449 flags.go:64] FLAG: --v="2"
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.129003    4449 flags.go:64] FLAG: --version="false"
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.129012    4449 flags.go:64] FLAG: --vmodule=""
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.129016    4449 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.129019    4449 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129204    4449 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129208    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129211    4449 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129228    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129230    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129234    4449 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129237    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129240    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129243    4449 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129245    4449 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129248    4449 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129251    4449 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:23.138296 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129253    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129256    4449 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129258    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129261    4449 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129263    4449 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129266    4449 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129269    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129271    4449 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129274    4449 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129277    4449 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129279    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129282    4449 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129284    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129287    4449 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129289    4449 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129292    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129294    4449 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129297    4449 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129305    4449 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129308    4449 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:23.138855 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129311    4449 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129314    4449 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129317    4449 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129323    4449 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129326    4449 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129330    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129333    4449 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129335    4449 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129338    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129340    4449 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129347    4449 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129350    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129353    4449 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129355    4449 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129358    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129360    4449 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129363    4449 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129365    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129368    4449 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129370    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:23.139345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129373    4449 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129375    4449 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129377    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129380    4449 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129382    4449 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129385    4449 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129387    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129390    4449 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129393    4449 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129395    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129398    4449 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129406    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129414    4449 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129417    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129419    4449 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129422    4449 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129424    4449 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129428    4449 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129431    4449 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129433    4449 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:23.139818 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129436    4449 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129438    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129442    4449 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129445    4449 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129448    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129450    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129452    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129455    4449 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129458    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129460    4449 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129463    4449 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129465    4449 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129468    4449 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129470    4449 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129473    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129477    4449 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129481    4449 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129484    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129487    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:23.140324 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129490    4449 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129492    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129495    4449 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129497    4449 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129500    4449 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129508    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129511    4449 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129513    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129516    4449 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129518    4449 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129522    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129524    4449 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129528    4449 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129531    4449 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129535    4449 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:23.140876 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.129563    4449 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129615    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129620    4449 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129623    4449 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129626    4449 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129629    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129632    4449 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129635    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129637    4449 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129640    4449 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129643    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129646    4449 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129648    4449 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129651    4449 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129653    4449 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129656    4449 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129658    4449 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129661    4449 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129663    4449 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129666    4449 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:23.141349 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129668    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129671    4449 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129673    4449 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129681    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129684    4449 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129686    4449 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129689    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129691    4449 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129695    4449 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129697    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129700    4449 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129702    4449 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129706    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129709    4449 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129711    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129714    4449 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129716    4449 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129718    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129721    4449 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:23.141884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129723    4449 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129726    4449 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129728    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129731    4449 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129733    4449 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129736    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129738    4449 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129740    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129743    4449 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129745    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129748    4449 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129750    4449 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129753    4449 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129755    4449 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129758    4449 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129760    4449 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129763    4449 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129772    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129774    4449 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129777    4449 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:23.142473 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129780    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129784    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129786    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129789    4449 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129798    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129802    4449 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129804    4449 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129807    4449 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129809    4449 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129812    4449 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129815    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129818    4449 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129822    4449 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129825    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129827    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129830    4449 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129832    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129835    4449 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129838    4449 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:23.143087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129840    4449 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129843    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129845    4449 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129848    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129850    4449 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129853    4449 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129855    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129858    4449 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129861    4449 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129864    4449 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129867    4449 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129875    4449 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129878    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129880    4449 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129884    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129887    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129889    4449 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129893    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129897    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:23.143591 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129899    4449 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129902    4449 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129904    4449 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129907    4449 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129909    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129912    4449 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129914    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129917    4449 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129919    4449 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.129921    4449 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.135530    4449 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.135547    4449 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.135564    4449 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.135570    4449 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135630    4449 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135636    4449 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135641    4449 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135644    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135648    4449 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:23.144084 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135651    4449 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135654    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135656    4449 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135659    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135662    4449 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135665    4449 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135667    4449 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135669    4449 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135673    4449 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135682    4449 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135685    4449 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135688    4449 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135691    4449 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135694    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135696    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135699    4449 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135702    4449 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135704    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135707    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:23.144578 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135710    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135713    4449 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135716    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135719    4449 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135722    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135724    4449 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135727    4449 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135730    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135733    4449 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135735    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135738    4449 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135741    4449 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135743    4449 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135746    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135748    4449 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135751    4449 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135753    4449 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135756    4449 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135760    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:23.145077 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135764    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135768    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135772    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135781    4449 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135784    4449 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135787    4449 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135790    4449 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135792    4449 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135795    4449 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135797    4449 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135800    4449 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135803    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135806    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135809    4449 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135812    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135814    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135817    4449 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135819    4449 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135822    4449 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135824    4449 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:23.145573 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135827    4449 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135830    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135832    4449 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135835    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135837    4449 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135840    4449 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135842    4449 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135845    4449 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135847    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135850    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135852    4449 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135855    4449 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135858    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135860    4449 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135863    4449 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135865    4449 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135869    4449 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135872    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135875    4449 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:23.146267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135878    4449 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135880    4449 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135883    4449 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135885    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135888    4449 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135891    4449 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135893    4449 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135896    4449 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135898    4449 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135901    4449 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135903    4449 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135906    4449 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135908    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135911    4449 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135913    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135916    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135918    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135921    4449 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135924    4449 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135926    4449 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:23.146967 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135929    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135931    4449 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135934    4449 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.135936    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.135963    4449 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136115    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136119    4449 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136122    4449 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136125    4449 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136128    4449 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136131    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136134    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136137    4449 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136139    4449 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136142    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136144    4449 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:23.147566 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136147    4449 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136149    4449 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136152    4449 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136154    4449 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136157    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136159    4449 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136162    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136164    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136167    4449 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136169    4449 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136172    4449 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136175    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136177    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136179    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136182    4449 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136184    4449 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136187    4449 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136189    4449 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136192    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136194    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:23.148247 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136197    4449 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136201    4449 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136205    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136208    4449 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136211    4449 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136234    4449 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136238    4449 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136241    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136244    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136246    4449 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136249    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136251    4449 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136254    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136256    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136259    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136261    4449 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136264    4449 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136266    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136269    4449 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:23.149026 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136272    4449 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136274    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136277    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136279    4449 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136281    4449 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136284    4449 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136287    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136289    4449 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136292    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136294    4449 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136297    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136300    4449 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136302    4449 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136305    4449 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136307    4449 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136310    4449 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136312    4449 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136315    4449 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136318    4449 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:23.149775 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136320    4449 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136323    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136326    4449 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136329    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136332    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136334    4449 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136336    4449 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136339    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136341    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136344    4449 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136346    4449 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136353    4449 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136355    4449 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136358    4449 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136360    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136363    4449 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136366    4449 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136368    4449 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136371    4449 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136373    4449 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:23.150371 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136376    4449 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136380    4449 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136384    4449 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136388    4449 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136392    4449 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136404    4449 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136407    4449 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136410    4449 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136412    4449 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136415    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136418    4449 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136420    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136423    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136425    4449 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136428    4449 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136432    4449 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:23.150885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136435    4449 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.136460    4449 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136492    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136496    4449 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136499    4449 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136503    4449 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136506    4449 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136509    4449 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136513    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136516    4449 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136519    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136521    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136524    4449 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136526    4449 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136529    4449 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136532    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:23.151292 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136534    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136537    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136539    4449 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136543    4449 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136546    4449 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136549    4449 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136552    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136554    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136557    4449 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136561    4449 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136564    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136567    4449 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136570    4449 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136572    4449 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136575    4449 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136577    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136580    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136583    4449 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136585    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:23.151674 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136588    4449 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136590    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136593    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136595    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136598    4449 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136601    4449 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136604    4449 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136607    4449 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136609    4449 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136612    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136614    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136617    4449 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136619    4449 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136622    4449 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136624    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136627    4449 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136633    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136636    4449 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136639    4449 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:23.152127 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136641    4449 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136644    4449 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136646    4449 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136649    4449 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136651    4449 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136654    4449 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136656    4449 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136659    4449 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136663    4449 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136665    4449 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136667    4449 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136670    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136673    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136676    4449 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136678    4449 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136681    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136683    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136686    4449 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136690    4449 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136692    4449 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:23.152666 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136695    4449 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136697    4449 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136700    4449 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136702    4449 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136705    4449 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136707    4449 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136710    4449 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136712    4449 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136715    4449 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136720    4449 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136723    4449 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136727    4449 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136730    4449 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136733    4449 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136736    4449 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136738    4449 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136741    4449 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136744    4449 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136746    4449 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:23.153147 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136749    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136751    4449 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136754    4449 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136756    4449 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136759    4449 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136763    4449 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136766    4449 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136768    4449 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136771    4449 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136774    4449 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136776    4449 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136780    4449 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136782    4449 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136785    4449 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:23.136787    4449 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.136843    4449 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.137808    4449 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.137861    4449 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 09:35:23.153748 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.138028    4449 server.go:1014] "Starting client certificate rotation"
Nov 02 09:35:23.154187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.138129    4449 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 09:35:23.154187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.138187    4449 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 09:29:42 +0000 UTC" deadline="2028-04-08 20:42:23.807645069 +0000 UTC"
Nov 02 09:35:23.154187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.138226    4449 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12563h7m0.66943368s"
Nov 02 09:35:23.154187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.138483    4449 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 09:35:23.154187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.139531    4449 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 09:35:23.154187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.141042    4449 log.go:25] "Validated CRI v1 runtime API"
Nov 02 09:35:23.154187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.142606    4449 log.go:25] "Validated CRI v1 image API"
Nov 02 09:35:23.154187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.143396    4449 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 09:35:23.154187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.144804    4449 fs.go:135] Filesystem UUIDs: map[2bab3e04-869d-40e0-abf0-80770041a7f4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d59d5d86-e009-4a82-8154-cfe9092d51b2:/dev/nvme0n1p4]
Nov 02 09:35:23.154588 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.144823    4449 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/069a9cd2953fdceae912f233fb6c8b15ddeeec78f0099252f640c71624b9e444/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/069a9cd2953fdceae912f233fb6c8b15ddeeec78f0099252f640c71624b9e444/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10e53b1d2a57c3d2d9d1e7dd55cd97e105ebbfcd12ffda01461a7d808689db63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10e53b1d2a57c3d2d9d1e7dd55cd97e105ebbfcd12ffda01461a7d808689db63/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/197ec8805706c766b2b8fc78d4600efcd2c70cbbda82317c9109581ed62b3b47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/197ec8805706c766b2b8fc78d4600efcd2c70cbbda82317c9109581ed62b3b47/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1cc09061c97b92a0cbebcd9d3aedb18b9004d60918452703e9c14d89714ae540/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1cc09061c97b92a0cbebcd9d3aedb18b9004d60918452703e9c14d89714ae540/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fce18bc9f01536ac7a0103974724a762893f0ddfaf69f8316ef71d73ee5b06f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fce18bc9f01536ac7a0103974724a762893f0ddfaf69f8316ef71d73ee5b06f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c827b8260d7b755e6d4866b18d443871095b032fc08bedfd6348b4fd0d77e1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c827b8260d7b755e6d4866b18d443871095b032fc08bedfd6348b4fd0d77e1c/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53fa6636a370d2cb041f3004e6a032772392ba78b6b30b8f49082e4d6a04b5ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53fa6636a370d2cb041f3004e6a032772392ba78b6b30b8f49082e4d6a04b5ce/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e3d93834a6f3ee2e9983e5648b508e4602ef6f81ecf374a7ec917fff5af183d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e3d93834a6f3ee2e9983e5648b508e4602ef6f81ecf374a7ec917fff5af183d/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6636d3d7808706e48bad250142326b9f47a56d1b998aa688a79e1bf2b97a77d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6636d3d7808706e48bad250142326b9f47a56d1b998aa688a79e1bf2b97a77d1/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bcc5a4312ea98f8d8dda276dd1dbe28ee676cb992f5cb49b90a023e42680e73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bcc5a4312ea98f8d8dda276dd1dbe28ee676cb992f5cb49b90a023e42680e73/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9f564fac29ec9b24b115967e1954028a16075c906a160c07daff681391c6e8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9f564fac29ec9b24b115967e1954028a16075c906a160c07daff681391c6e8e/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8e16647170d3db67b1a128ab9a6baabae43570fa576409e400e59bb6d236856/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8e16647170d3db67b1a128ab9a6baabae43570fa576409e400e59bb6d236856/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d13ce16fb67c2d2ded6fb06c09753f0bcbde3a067663ef5d3641dbc97fc69783/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d13ce16fb67c2d2ded6fb06c09753f0bcbde3a067663ef5d3641dbc97fc69783/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d912ba6609f62041b568e2fd0d24f5c414b04c4d859a3463397de5032b0b3d97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d912ba6609f62041b568e2fd0d24f5c414b04c4d859a3463397de5032b0b3d97/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee5f0177d6700e6cbf38f06cb2133bcdddbf0610bf4374dacd37a29d58417b34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee5f0177d6700e6cbf38f06cb2133bcdddbf0610bf4374dacd37a29d58417b34/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923/volumes/kubernetes.io~projected/kube-api-access-h9sfz:{mountpoint:/var/lib/kubelet/pods/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923/volumes/kubernetes.io~projected/kube-api-access-h9sfz major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10db22d0-6805-4d79-a670-e7b96a3b9c22/volumes/kubernetes.io~projected/kube-api-access-j7bjq:{mountpoint:/var/lib/kubelet/pods/10db22d0-6805-4d79-a670-e7b96a3b9c22/volumes/kubernetes.io~projected/kube-api-access-j7bjq major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18e3a7af-aafb-4491-b696-95f4858b585a/volumes/kubernetes.io~projected/kube-api-access-5fqq6:{mountpoint:/var/lib/kubelet/pods/18e3a7af-aafb-4491-b696-95f4858b585a/volumes/kubernetes.io~projected/kube-api-access-5fqq6 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ee8e64e-98c3-442e-9885-f45afd29a812/volumes/kubernetes.io~projected/kube-api-access-sl5qh:{mountpoint:/var/lib/kubelet/pods/1ee8e64e-98c3-442e-9885-f45afd29a812/volumes/kubernetes.io~projected/kube-api-access-sl5qh major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/260f9ad0-a637-4423-ac4c-626c79e6bca9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/260f9ad0-a637-4423-ac4c-626c79e6bca9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/260f9ad0-a637-4423-ac4c-626c79e6bca9/volumes/kubernetes.io~projected/kube-api-access-r5qq5:{mountpoint:/var/lib/kubelet/pods/260f9ad0-a637-4423-ac4c-626c79e6bca9/volumes/kubernetes.io~projected/kube-api-access-r5qq5 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/260f9ad0-a637-4423-ac4c-626c79e6bca9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/260f9ad0-a637-4423-ac4c-626c79e6bca9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50bebf56-2ddc-47e0-b8f7-008d202e3af3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/50bebf56-2ddc-47e0-b8f7-008d202e3af3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50bebf56-2ddc-47e0-b8f7-008d202e3af3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/50bebf56-2ddc-47e0-b8f7-008d202e3af3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50bebf56-2ddc-47e0-b8f7-008d202e3af3/volumes/kubernetes.io~projected/kube-api-access-9nf7t:{mountpoint:/var/lib/kubelet/pods/50bebf56-2ddc-47e0-b8f7-008d202e3af3/volumes/kubernetes.io~projected/kube-api-access-9nf7t major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cf1aef2-0bc5-445e-880d-90215888057a/volumes/kubernetes.io~projected/kube-api-access-6xbhc:{mountpoint:/var/lib/kubelet/pods/5cf1aef2-0bc5-445e-880d-90215888057a/volumes/kubernetes.io~projected/kube-api-access-6xbhc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60edc0c2-d978-40b5-b155-152019a3bdf5/volumes/kubernetes.io~projected/kube-api-access-wwnd7:{mountpoint:/var/lib/kubelet/pods/60edc0c2-d978-40b5-b155-152019a3bdf5/volumes/kubernetes.io~projected/kube-api-access-wwnd7 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69f3517a-6eb0-4b5a-a5de-fa56577626be/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/69f3517a-6eb0-4b5a-a5de-fa56577626be/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86739438-2924-4642-9d34-ed545e1739ba/volumes/kubernetes.io~projected/kube-api-access-lsmrc:{mountpoint:/var/lib/kubelet/pods/86739438-2924-4642-9d34-ed545e1739ba/volumes/kubernetes.io~projected/kube-api-access-lsmrc major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e9ee3cd-e542-40a1-a18f-2843dcd82a69/volumes/kubernetes.io~projected/kube-api-access-gffwf:{mountpoint:/var/lib/kubelet/pods/8e9ee3cd-e542-40a1-a18f-2843dcd82a69/volumes/kubernetes.io~projected/kube-api-access-gffwf major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0a20d22-7ea2-4cb8-a46f-d48043889d6f/volumes/kubernetes.io~projected/kube-api-access-lbngm:{mountpoint:/var/lib/kubelet/pods/b0a20d22-7ea2-4cb8-a46f-d48043889d6f/volumes/kubernetes.io~projected/kube-api-access-lbngm major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0a20d22-7ea2-4cb8-a46f-d48043889d6f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b0a20d22-7ea2-4cb8-a46f-d48043889d6f/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c108bc48-9b97-46d1-a353-04a6dbc9ea83/volumes/kubernetes.io~projected/kube-api-access-lptr9:{mountpoint:/var/lib/kubelet/pods/c108bc48-9b97-46d1-a353-04a6dbc9ea83/volumes/kubernetes.io~projected/kube-api-access-lptr9 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd36afed-d103-4f7a-ba99-00c841bb9537/volumes/kubernetes.io~projected/kube-api-access-rvstx:{mountpoint:/var/lib/kubelet/pods/cd36afed-d103-4f7a-ba99-00c841bb9537/volumes/kubernetes.io~projected/kube-api-access-rvstx major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa/volumes/kubernetes.io~projected/kube-api-access-wm85w:{mountpoint:/var/lib/kubelet/pods/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa/volumes/kubernetes.io~projected/kube-api-access-wm85w major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef320c69-127d-437a-87ed-cfcd9db66325/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ef320c69-127d-437a-87ed-cfcd9db66325/volumes/kubernetes.io~secret/agent-certs major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab/volumes/kubernetes.io~projected/kube-api-access-ttf7t:{mountpoint:/var/lib/kubelet/pods/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab/volumes/kubernetes.io~projected/kube-api-access-ttf7t major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d25031ae18a35d113255b709b5cc4e651a27409e5df7b3fd542dfe84447feb83/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/af964ad01706b7a82e659f23b47fa99bc7ba85420d8fa2d6e92c50a5d5c13f86/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/9aff7f86753a9461dfbd21f19e3ea812c0bd15daa2430ce3c13971f79681b571/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/0b1ce9de3ceb3dfdbd5efae62c24715efee7790833065ecc553551ea61fb96ca/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0c46e7418dce3d577fef4061a0a23dd341ee72335b002d3ab7f11d28f7c9d975/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3dd21b475d547e6da86b1c3e621f75fc69ff947d1336744d6fe04df37ae68ae6/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9f3a2ad60d4bb5ceb2d7ba5ce78aa30dd61b4767e7c136e63ef122de555d6b38/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/99b9cdb2b0bf0a44012c8e52e1c2037999e455b7fb054176384ef498d353e4d9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/061a088b24733c06ed12cabe24e2211b2a6b0e53f1c9168594077716821a552f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/7ef42c79b110cd10251e64a39f452d82d83d1167e95f035c8d2eab458321d285/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/a2f9ab5d2bc1b32e231ea4df411d68c328299801e55aad00a7e06e51cba0aa49/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/64a680cec188a1d340f9525734c7e397e560a6765ced8216eef3423dfdc4b712/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/b3e262364507aa90bb474781407834b2a519f4e8b1519300acbf2221bb3b5c8b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ee71ec3a1c7b67668c609faa2d0c32de5322a8ac966bfe2449d016c3c1187da8/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/cbb66df3570dc54d44ee96fbcf450ab11b1cbbd33690544027cf5abfde7cd4ad/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d6c76cc68fbf3114de3574c0a4ac1672228f92c6f6d64ed9c4b5cd1f903592db/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/06aceda0cc97671703f24a33ae2058a409ddf03ec37a27967f1f2d9978dd367c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/04362ef6a30f3ddcfa95a20d582aa473cd94fabd7c0eedd438a50586c21a90b2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/bf103fbedf7744342f8d44766da2721fa8507418d0cb6ed020b245c3431488d1/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0f999281210d9bbd19838ce8acd47d0af301222c180fedf7bb72f242df1949f2/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/41bc6ca1dce75c47f4921b26a22c9e4ed3b9c3fa2c8e772d9976bd65ead0a7a0/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/febc6690e950b67d8fa9f64f90710964c970535274f2fc9c6dd395881668adc9/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/51d6c0644630bb02d836db66f419cc6e7b47c8b8e293d549c184e7bf9f5d5c7b/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/5d3a3e9f4a32c0c4709359cf85f19316dc6e304da60c2cc9b19d43899cccaf54/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/afc87e042235e412e7838d518801f5607c46fc64f6f6096436ddcfe4b3f21783/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/21139e400ae65c50ac65b62915686f151f75269323245e61d3c83fe264461127/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/6024ef022b4829982b122c4b403185e6e9dfd5f3c0911468ddbc3720809eb1bb/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/6114c2853d4d454a0b9b14f270fb94821e80c34a527c1d8e65e64b9e3f417912/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/6487c42cc3e28608b9165d9a01cd963cacd0ee2e0a5d43d4208d32bdf7c98a80/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/b7596ce9ea8e1aca9e93fe3e4f0415cef532b31769e893e9004bf479a7b1ca4e/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/7abf8eb44feb2ada1dd18d7881ea827398b93be30c416f5839cbc8d5147caf91/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/e56f5138e5b419500b3d93516232189dc06b33a984fe9637a6aba1af079be5b1/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f2a7c20216c7e34d1fa5146c36e1285044d33ba9d9e4ea77af8810ce98c5f1fe/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/548f204ca0a4b63c7b6b717bb91d7874c3ba31f37f6acbd0dce12d5653b9b84c/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a01946c63fd5f106eb918b17548248267b04ff004273a38729776a218c77e896/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/0a1c20d965e0488d5832eb7ff85a58a976767cc38174bcb7caf77ff213cd5bf2/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/754de0904a9acd8cb1efe616b2bb2348f0a96c94a1247326d32b95b51bbfc483/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/96dd417f840181b11f34b1f72b4bb473b53ef2ef28d5a2a1fd14902ed8fae5a9/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8a8e5080643eac12b96232661725599ba84ea853c4f22090752c7035aed15fc3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/18ffc9d87fb6f247493d6e9fb508cf1f8c1d62a11208d786e1cb15c82fc1e203/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/fb04ca8da2ab8db6da4cc38c8a2b343e04b81b6844affa752311bb1a4e541ecf/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.152170    4449 manager.go:217] Machine: {Timestamp:2026-11-02 09:35:23.151728559 +0000 UTC m=+0.090015325 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095870 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d664e7afd2bf906b0bae40b8bcfbc SystemUUID:ec2d664e-7afd-2bf9-06b0-bae40b8bcfbc BootID:b4c7187c-a3b3-472c-b272-280d026c5336 Filesystems:[{Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d13ce16fb67c2d2ded6fb06c09753f0bcbde3a067663ef5d3641dbc97fc69783/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa/volumes/kubernetes.io~projected/kube-api-access-wm85w DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10e53b1d2a57c3d2d9d1e7dd55cd97e105ebbfcd12ffda01461a7d808689db63/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0a20d22-7ea2-4cb8-a46f-d48043889d6f/volumes/kubernetes.io~projected/kube-api-access-lbngm DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c827b8260d7b755e6d4866b18d443871095b032fc08bedfd6348b4fd0d77e1c/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/260f9ad0-a637-4423-ac4c-626c79e6bca9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60edc0c2-d978-40b5-b155-152019a3bdf5/volumes/kubernetes.io~projected/kube-api-access-wwnd7 DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d912ba6609f62041b568e2fd0d24f5c414b04c4d859a3463397de5032b0b3d97/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53fa6636a370d2cb041f3004e6a032772392ba78b6b30b8f49082e4d6a04b5ce/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50bebf56-2ddc-47e0-b8f7-008d202e3af3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/069a9cd2953fdceae912f233fb6c8b15ddeeec78f0099252f640c71624b9e444/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c108bc48-9b97-46d1-a353-04a6dbc9ea83/volumes/kubernetes.io~projected/kube-api-access-lptr9 DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1ee8e64e-98c3-442e-9885-f45afd29a812/volumes/kubernetes.io~projected/kube-api-access-sl5qh DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9f564fac29ec9b24b115967e1954028a16075c906a160c07daff681391c6e8e/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10db22d0-6805-4d79-a670-e7b96a3b9c22/volumes/kubernetes.io~projected/kube-api-access-j7bjq DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee5f0177d6700e6cbf38f06cb2133bcdddbf0610bf4374dacd37a29d58417b34/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0a20d22-7ea2-4cb8-a46f-d48043889d6f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/69f3517a-6eb0-4b5a-a5de-fa56577626be/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6636d3d7808706e48bad250142326b9f47a56d1b998aa688a79e1bf2b97a77d1/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e9ee3cd-e542-40a1-a18f-2843dcd82a69/volumes/kubernetes.io~projected/kube-api-access-gffwf DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/260f9ad0-a637-4423-ac4c-626c79e6bca9/volumes/kubernetes.io~projected/kube-api-access-r5qq5 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/197ec8805706c766b2b8fc78d4600efcd2c70cbbda82317c9109581ed62b3b47/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8e16647170d3db67b1a128ab9a6baabae43570fa576409e400e59bb6d236856/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1cc09061c97b92a0cbebcd9d3aedb18b9004d60918452703e9c14d89714ae540/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e3d93834a6f3ee2e9983e5648b508e4602ef6f81ecf374a7ec917fff5af183d/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/260f9ad0-a637-4423-ac4c-626c79e6bca9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18e3a7af-aafb-4491-b696-95f4858b585a/volumes/kubernetes.io~projected/kube-api-access-5fqq6 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ef320c69-127d-437a-87ed-cfcd9db66325/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cd36afed-d103-4f7a-ba99-00c841bb9537/volumes/kubernetes.io~projected/kube-api-access-rvstx DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/50bebf56-2ddc-47e0-b8f7-008d202e3af3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/50bebf56-2ddc-47e0-b8f7-008d202e3af3/volumes/kubernetes.io~projected/kube-api-access-9nf7t DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fce18bc9f01536ac7a0103974724a762893f0ddfaf69f8316ef71d73ee5b06f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/5cf1aef2-0bc5-445e-880d-90215888057a/volumes/kubernetes.io~projected/kube-api-access-6xbhc DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab/volumes/kubernetes.io~projected/kube-api-access-ttf7t DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923/volumes/kubernetes.io~projected/kube-api-access-h9sfz DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bcc5a4312ea98f8d8dda276dd1dbe28ee676cb992f5cb49b90a023e42680e73/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86739438-2924-4642-9d34-ed545e1739ba/volumes/kubernetes.io~projected/kube-api-access-lsmrc DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1cc09061c97b92a MacAddress:06:fe:ba:81:c3:e1 Speed:10000 Mtu:8901} {Name:2c827b8260d7b75 MacAddress:1a:e5:72:5d:cc:dd Speed:10000 Mtu:8901} {Name:6636d3d7808706e MacAddress:e2:48:7c:ee:9a:db Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e9:6f:79:81 Speed:0 Mtu:9001} {Name:br-int MacAddress:8e:8b:ed:03:c0:4a Speed:0 Mtu:8901} {Name:d13ce16fb67c2d2 MacAddress:72:04:41:83:96:8d Speed:10000 Mtu:8901} {Name:d912ba6609f6204 MacAddress:86:dd:84:31:7b:9e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e9:6f:79:81 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:52:8c:cf:47:3f:e7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3e:a9:4d:7a:10:09 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.152441    4449 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.152484    4449 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.152714    4449 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.152729    4449 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-217.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.152842    4449 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.152849    4449 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.152862    4449 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.152880    4449 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.152951    4449 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.153096    4449 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.153210    4449 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.153265    4449 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.153278    4449 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.153288    4449 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.153296    4449 kubelet.go:405] "Adding apiserver pod source"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.153307    4449 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.154010    4449 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.154024    4449 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 09:35:23.154792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.154630    4449 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 09:35:23.156755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.156739    4449 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 09:35:23.156755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.156754    4449 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 09:35:23.156878 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.156868    4449 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 09:35:23.157053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157042    4449 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 09:35:23.157082 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157072    4449 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 09:35:23.157173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157164    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 09:35:23.157202 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157178    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 09:35:23.157202 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157185    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 09:35:23.157202 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157191    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 09:35:23.157202 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157199    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 09:35:23.157335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157208    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 09:35:23.157335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157232    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 09:35:23.157335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157239    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 09:35:23.157335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157246    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 09:35:23.157335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157252    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 09:35:23.157335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157270    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 09:35:23.157335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157282    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 09:35:23.157335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157311    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 09:35:23.157335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.157317    4449 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 09:35:23.160633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.160618    4449 server.go:1263] "Started kubelet"
Nov 02 09:35:23.161166 ip-10-0-128-217 systemd[1]: Started Kubernetes Kubelet.
Nov 02 09:35:23.161645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.160697    4449 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 09:35:23.162193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.160738    4449 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 09:35:23.164958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.162740    4449 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 09:35:23.164958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.163179    4449 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 09:35:23.164958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.163916    4449 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 09:35:23.164958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.164694    4449 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 09:35:23.165670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.165650    4449 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 09:35:23.165852 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.165811    4449 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 09:29:44 +0000 UTC" deadline="2028-08-04 16:35:13.30978086 +0000 UTC"
Nov 02 09:35:23.165931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.165854    4449 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15390h59m50.143933258s"
Nov 02 09:35:23.165931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.165890    4449 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 09:35:23.165931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.165905    4449 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 09:35:23.166071 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.166012    4449 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 09:35:23.166147 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:23.166127    4449 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-217.us-east-2.compute.internal\" not found"
Nov 02 09:35:23.168130 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168100    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd36afed-d103-4f7a-ba99-00c841bb9537" volumeName="kubernetes.io/projected/cd36afed-d103-4f7a-ba99-00c841bb9537-kube-api-access-rvstx" seLinuxMountContext=""
Nov 02 09:35:23.168130 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168129    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fab1d8c2-b3fa-43bc-9009-a0a9c3079eab" volumeName="kubernetes.io/configmap/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-cni-binary-copy" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168140    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fab1d8c2-b3fa-43bc-9009-a0a9c3079eab" volumeName="kubernetes.io/projected/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-kube-api-access-ttf7t" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168148    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18e3a7af-aafb-4491-b696-95f4858b585a" volumeName="kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168156    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50bebf56-2ddc-47e0-b8f7-008d202e3af3" volumeName="kubernetes.io/empty-dir/50bebf56-2ddc-47e0-b8f7-008d202e3af3-tmp" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168164    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86739438-2924-4642-9d34-ed545e1739ba" volumeName="kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168171    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c108bc48-9b97-46d1-a353-04a6dbc9ea83" volumeName="kubernetes.io/empty-dir/c108bc48-9b97-46d1-a353-04a6dbc9ea83-data-volume" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168178    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ee8e64e-98c3-442e-9885-f45afd29a812" volumeName="kubernetes.io/projected/1ee8e64e-98c3-442e-9885-f45afd29a812-kube-api-access-sl5qh" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168188    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0a20d22-7ea2-4cb8-a46f-d48043889d6f" volumeName="kubernetes.io/secret/b0a20d22-7ea2-4cb8-a46f-d48043889d6f-cert" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168195    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef320c69-127d-437a-87ed-cfcd9db66325" volumeName="kubernetes.io/configmap/ef320c69-127d-437a-87ed-cfcd9db66325-konnectivity-ca" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168202    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0556aaf2-dc7e-4e57-a721-4e0ee6cfd923" volumeName="kubernetes.io/projected/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-kube-api-access-h9sfz" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168236    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18e3a7af-aafb-4491-b696-95f4858b585a" volumeName="kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168243    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50bebf56-2ddc-47e0-b8f7-008d202e3af3" volumeName="kubernetes.io/empty-dir/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-tuned" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168250    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50bebf56-2ddc-47e0-b8f7-008d202e3af3" volumeName="kubernetes.io/projected/50bebf56-2ddc-47e0-b8f7-008d202e3af3-kube-api-access-9nf7t" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168262    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69f3517a-6eb0-4b5a-a5de-fa56577626be" volumeName="kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret" seLinuxMountContext=""
Nov 02 09:35:23.168284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168281    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0a20d22-7ea2-4cb8-a46f-d48043889d6f" volumeName="kubernetes.io/projected/b0a20d22-7ea2-4cb8-a46f-d48043889d6f-kube-api-access-lbngm" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168292    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c108bc48-9b97-46d1-a353-04a6dbc9ea83" volumeName="kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168302    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10db22d0-6805-4d79-a670-e7b96a3b9c22" volumeName="kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168313    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="260f9ad0-a637-4423-ac4c-626c79e6bca9" volumeName="kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-env-overrides" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168323    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="260f9ad0-a637-4423-ac4c-626c79e6bca9" volumeName="kubernetes.io/secret/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168335    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86739438-2924-4642-9d34-ed545e1739ba" volumeName="kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-service-ca-bundle" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168346    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef320c69-127d-437a-87ed-cfcd9db66325" volumeName="kubernetes.io/secret/ef320c69-127d-437a-87ed-cfcd9db66325-agent-certs" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168356    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fab1d8c2-b3fa-43bc-9009-a0a9c3079eab" volumeName="kubernetes.io/configmap/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-daemon-config" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168367    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86739438-2924-4642-9d34-ed545e1739ba" volumeName="kubernetes.io/empty-dir/86739438-2924-4642-9d34-ed545e1739ba-archives-path" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168377    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e9ee3cd-e542-40a1-a18f-2843dcd82a69" volumeName="kubernetes.io/empty-dir/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-tmp-dir" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168387    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e9ee3cd-e542-40a1-a18f-2843dcd82a69" volumeName="kubernetes.io/projected/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-kube-api-access-gffwf" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168408    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="260f9ad0-a637-4423-ac4c-626c79e6bca9" volumeName="kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168420    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="260f9ad0-a637-4423-ac4c-626c79e6bca9" volumeName="kubernetes.io/projected/260f9ad0-a637-4423-ac4c-626c79e6bca9-kube-api-access-r5qq5" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168436    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60edc0c2-d978-40b5-b155-152019a3bdf5" volumeName="kubernetes.io/configmap/60edc0c2-d978-40b5-b155-152019a3bdf5-config-volume" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168447    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60edc0c2-d978-40b5-b155-152019a3bdf5" volumeName="kubernetes.io/projected/60edc0c2-d978-40b5-b155-152019a3bdf5-kube-api-access-wwnd7" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168458    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa" volumeName="kubernetes.io/projected/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-kube-api-access-wm85w" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168470    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cf1aef2-0bc5-445e-880d-90215888057a" volumeName="kubernetes.io/configmap/5cf1aef2-0bc5-445e-880d-90215888057a-iptables-alerter-script" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168482    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86739438-2924-4642-9d34-ed545e1739ba" volumeName="kubernetes.io/projected/86739438-2924-4642-9d34-ed545e1739ba-kube-api-access-lsmrc" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168492    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c108bc48-9b97-46d1-a353-04a6dbc9ea83" volumeName="kubernetes.io/projected/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-api-access-lptr9" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168504    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60edc0c2-d978-40b5-b155-152019a3bdf5" volumeName="kubernetes.io/empty-dir/60edc0c2-d978-40b5-b155-152019a3bdf5-tmp-dir" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168515    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0556aaf2-dc7e-4e57-a721-4e0ee6cfd923" volumeName="kubernetes.io/configmap/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-serviceca" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168526    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18e3a7af-aafb-4491-b696-95f4858b585a" volumeName="kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-cni-binary-copy" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168536    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18e3a7af-aafb-4491-b696-95f4858b585a" volumeName="kubernetes.io/projected/18e3a7af-aafb-4491-b696-95f4858b585a-kube-api-access-5fqq6" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168547    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="260f9ad0-a637-4423-ac4c-626c79e6bca9" volumeName="kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovnkube-config" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168559    4449 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cf1aef2-0bc5-445e-880d-90215888057a" volumeName="kubernetes.io/projected/5cf1aef2-0bc5-445e-880d-90215888057a-kube-api-access-6xbhc" seLinuxMountContext=""
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168571    4449 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 09:35:23.168953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.168578    4449 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 09:35:23.170115 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.170099    4449 factory.go:55] Registering systemd factory
Nov 02 09:35:23.170115 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.170117    4449 factory.go:223] Registration of the systemd container factory successfully
Nov 02 09:35:23.173823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.172192    4449 factory.go:153] Registering CRI-O factory
Nov 02 09:35:23.173823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.172210    4449 factory.go:223] Registration of the crio container factory successfully
Nov 02 09:35:23.173823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.172289    4449 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 09:35:23.173823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.172307    4449 factory.go:103] Registering Raw factory
Nov 02 09:35:23.173823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.172321    4449 manager.go:1199] Started watching for new ooms in manager
Nov 02 09:35:23.173823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.172677    4449 manager.go:319] Starting recovery of all containers
Nov 02 09:35:23.174120 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.174023    4449 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:35:23.175645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.175625    4449 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:35:23.176180 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.176163    4449 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:35:23.176324 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.176311    4449 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 09:35:23.196975 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.196944    4449 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 09:35:23.198387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.198366    4449 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 09:35:23.198457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.198396    4449 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 09:35:23.198457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.198414    4449 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 09:35:23.198536 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:23.198444    4449 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 09:35:23.202157 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.202140    4449 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:35:23.210012 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.209988    4449 generic.go:356] "Generic (PLEG): container finished" podID="7bc16cf3ad519d4529637b03314e6bd7" containerID="b9d0478422abfeb70f1ac0e4ef1d1d0f50c3d7b7383fa494c19cc92504eeae8e" exitCode=0
Nov 02 09:35:23.220018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.219997    4449 generic.go:356] "Generic (PLEG): container finished" podID="18e3a7af-aafb-4491-b696-95f4858b585a" containerID="e348a68294de81a65c9e7b5ba2b75510d4a6d84c8c84018d2fbd7c5c0a889bb2" exitCode=0
Nov 02 09:35:23.220018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.220018    4449 generic.go:356] "Generic (PLEG): container finished" podID="18e3a7af-aafb-4491-b696-95f4858b585a" containerID="4b88618acde430f4002965370f2a88d17bc57f166a3f24dd3e20a4f5678e67c7" exitCode=0
Nov 02 09:35:23.220161 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.220027    4449 generic.go:356] "Generic (PLEG): container finished" podID="18e3a7af-aafb-4491-b696-95f4858b585a" containerID="e3c8617964cce559b85b2da5f469dbb7ebee7e00667e36c9d75c9620dd437a6e" exitCode=0
Nov 02 09:35:23.220161 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.220036    4449 generic.go:356] "Generic (PLEG): container finished" podID="18e3a7af-aafb-4491-b696-95f4858b585a" containerID="b5e937b028cd46cc80f79dd19de80161d719a54763a0fa3190d629ae72233627" exitCode=0
Nov 02 09:35:23.220161 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.220046    4449 generic.go:356] "Generic (PLEG): container finished" podID="18e3a7af-aafb-4491-b696-95f4858b585a" containerID="f539a73f4ce7ddf87b392003d5b6a572f8d5cc0b0f40618bdc8eba663dedffb0" exitCode=0
Nov 02 09:35:23.220161 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.220053    4449 generic.go:356] "Generic (PLEG): container finished" podID="18e3a7af-aafb-4491-b696-95f4858b585a" containerID="c04d27bb4cfc72e4d277593d79f3b2402596b919ec90eb242f951cbd0d9a86a0" exitCode=0
Nov 02 09:35:23.239497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.239479    4449 manager.go:324] Recovery completed
Nov 02 09:35:23.254920 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.254898    4449 cpu_manager.go:226] "Starting" policy="none"
Nov 02 09:35:23.254920 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.254913    4449 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 09:35:23.255073 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.254934    4449 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 09:35:23.255133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.255077    4449 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 09:35:23.255133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.255094    4449 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 09:35:23.255133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.255117    4449 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 09:35:23.255133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.255126    4449 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 09:35:23.255133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.255134    4449 policy_none.go:50] "Start"
Nov 02 09:35:23.255382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.255147    4449 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 09:35:23.255382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.255161    4449 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 09:35:23.255382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.255369    4449 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 09:35:23.255382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.255381    4449 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 09:35:23.255555 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.255389    4449 policy_none.go:44] "Start"
Nov 02 09:35:23.259283 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.259197    4449 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 09:35:23.264461 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.264447    4449 manager.go:341] "Starting Device Plugin manager"
Nov 02 09:35:23.265118 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:23.264478    4449 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 09:35:23.265171 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.265123    4449 server.go:87] "Starting device plugin registration server"
Nov 02 09:35:23.265459 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.265447    4449 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 09:35:23.265491 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.265463    4449 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 09:35:23.265572 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.265556    4449 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 09:35:23.265680 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.265643    4449 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 09:35:23.265680 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.265677    4449 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 09:35:23.265790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.265684    4449 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 09:35:23.265790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.265719    4449 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T09:35:23.265667131Z","UUID":"2cb0965c-e2eb-4736-adff-6c840dae6929","Handler":null,"Name":"","Endpoint":""}
Nov 02 09:35:23.266816 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:23.266798    4449 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 09:35:23.266945 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.266929    4449 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 09:35:23.266945 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.266953    4449 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 09:35:23.299233 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.299183    4449 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"]
Nov 02 09:35:23.299657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.299628    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal" event={"ID":"7bc16cf3ad519d4529637b03314e6bd7","Type":"ContainerStarted","Data":"2a599f32843847a927e0cdc87b580ddf850cb412b6c34639cd66422f1ff79766"}
Nov 02 09:35:23.299732 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.299666    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal" event={"ID":"7bc16cf3ad519d4529637b03314e6bd7","Type":"ContainerDied","Data":"b9d0478422abfeb70f1ac0e4ef1d1d0f50c3d7b7383fa494c19cc92504eeae8e"}
Nov 02 09:35:23.299732 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.299715    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal" event={"ID":"7bc16cf3ad519d4529637b03314e6bd7","Type":"ContainerStarted","Data":"10e53b1d2a57c3d2d9d1e7dd55cd97e105ebbfcd12ffda01461a7d808689db63"}
Nov 02 09:35:23.299832 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.299813    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal" event={"ID":"89488fe67d46e24a4bfb2dbd39ca56e0","Type":"ContainerStarted","Data":"27ca94c5429d79d4708706c4c42cfcdf8edae3c7d2eda17e00f13f1919d82753"}
Nov 02 09:35:23.299832 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.299823    4449 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.299997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.299840    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal" event={"ID":"89488fe67d46e24a4bfb2dbd39ca56e0","Type":"ContainerStarted","Data":"ee5f0177d6700e6cbf38f06cb2133bcdddbf0610bf4374dacd37a29d58417b34"}
Nov 02 09:35:23.299997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.299905    4449 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.310135 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.310119    4449 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 09:35:23.310197 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:23.310169    4449 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.310809 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.310795    4449 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 09:35:23.310861 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:23.310829    4449 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.365940 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.365915    4449 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:35:23.367009 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.366994    4449 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:35:23.367077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.367023    4449 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:35:23.367077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.367034    4449 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-217.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:35:23.367176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.367078    4449 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.377024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.377004    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7bc16cf3ad519d4529637b03314e6bd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"7bc16cf3ad519d4529637b03314e6bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.377081 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.377029    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bc16cf3ad519d4529637b03314e6bd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"7bc16cf3ad519d4529637b03314e6bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.377081 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.377053    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89488fe67d46e24a4bfb2dbd39ca56e0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"89488fe67d46e24a4bfb2dbd39ca56e0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.380380 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.380366    4449 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.380437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.380425    4449 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.477302 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.477276    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89488fe67d46e24a4bfb2dbd39ca56e0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"89488fe67d46e24a4bfb2dbd39ca56e0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.477302 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.477304    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7bc16cf3ad519d4529637b03314e6bd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"7bc16cf3ad519d4529637b03314e6bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.477467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.477321    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bc16cf3ad519d4529637b03314e6bd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"7bc16cf3ad519d4529637b03314e6bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.477467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.477362    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bc16cf3ad519d4529637b03314e6bd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"7bc16cf3ad519d4529637b03314e6bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.477467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.477404    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89488fe67d46e24a4bfb2dbd39ca56e0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"89488fe67d46e24a4bfb2dbd39ca56e0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:23.477467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:23.477407    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7bc16cf3ad519d4529637b03314e6bd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" (UID: \"7bc16cf3ad519d4529637b03314e6bd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:24.154125 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.154095    4449 apiserver.go:52] "Watching apiserver"
Nov 02 09:35:24.174691 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.174664    4449 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 09:35:24.175126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.175107    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-p8bj8","openshift-multus/multus-vkr2t","openshift-image-registry/node-ca-hxwln","openshift-insights/insights-runtime-extractor-smlj2","kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-8pbvj","openshift-dns/dns-default-sjj6v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd","openshift-cluster-node-tuning-operator/tuned-jxnjv","openshift-ovn-kubernetes/ovnkube-node-px4ck","kube-system/global-pull-secret-syncer-8w4cb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal","kube-system/konnectivity-agent-h48j7","openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz","openshift-insights/periodic-gathering-pgbtq-2c6rz","openshift-ingress-canary/ingress-canary-l7v8r","openshift-network-operator/iptables-alerter-r8j97","openshift-network-diagnostics/network-check-target-2gvpb","openshift-dns/node-resolver-r85rg"]
Nov 02 09:35:24.179072 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.179052    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:24.179946 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.179726    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f2j22\""
Nov 02 09:35:24.181059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.180556    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 09:35:24.181059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.180741    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 09:35:24.181059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.180893    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 09:35:24.181059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.181027    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qbp4s\""
Nov 02 09:35:24.182843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.181371    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sbddj\""
Nov 02 09:35:24.182843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.181561    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kdt4x\""
Nov 02 09:35:24.182843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.181931    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:24.182843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.182456    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9d4cl\""
Nov 02 09:35:24.182843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.182479    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 09:35:24.182843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.182485    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 09:35:24.182843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.182688    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t9b8s\""
Nov 02 09:35:24.182843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.182718    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbhf4\""
Nov 02 09:35:24.183261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.182867    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-982vl\""
Nov 02 09:35:24.183261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.182934    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-llzhk\""
Nov 02 09:35:24.183375 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.183356    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zc5hs\""
Nov 02 09:35:24.183907 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.183890    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7vqld\""
Nov 02 09:35:24.183968 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.183911    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 09:35:24.184016 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.183975    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 09:35:24.184075 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184020    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 09:35:24.184075 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184052    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 09:35:24.184171 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184085    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 09:35:24.184171 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184117    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 09:35:24.184171 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184126    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 09:35:24.184326 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184181    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 09:35:24.184326 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184301    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 09:35:24.184531 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184513    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9tqmn\""
Nov 02 09:35:24.184653 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184635    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 09:35:24.184809 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184772    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 09:35:24.184933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184831    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 09:35:24.184933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184841    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 09:35:24.184933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184852    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 09:35:24.184933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184878    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:24.184933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184889    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 09:35:24.184933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184897    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 09:35:24.184933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184929    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 09:35:24.185392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184929    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 09:35:24.185392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184954    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 09:35:24.185392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.184976    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 09:35:24.185392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.185006    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 09:35:24.185392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.185020    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:24.185392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.185050    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 09:35:24.185392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.185051    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:24.185392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.185094    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:24.185392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.185096    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:24.185392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.185200    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 09:35:24.185392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.185200    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 09:35:24.185392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.185301    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 09:35:24.185392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.185320    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 09:35:24.185914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.185843    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gztjj\""
Nov 02 09:35:24.185914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.185860    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 09:35:24.185914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.185864    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zwj6d\""
Nov 02 09:35:24.188771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.188754    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xrsh4\""
Nov 02 09:35:24.188879 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.188861    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fzpc9\""
Nov 02 09:35:24.188879 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.188872    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 09:35:24.188987 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.188978    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6hxzv\""
Nov 02 09:35:24.189319 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.189306    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 09:35:24.189411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.189335    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:24.189411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.189391    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 09:35:24.189573 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.189425    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 09:35:24.189573 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.189432    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 09:35:24.189573 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.189505    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 09:35:24.192067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.192052    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 09:35:24.237200 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.237182    4449 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:24.237337 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.237324    4449 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:24.247391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.247371    4449 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 09:35:24.247471 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.247407    4449 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:24.248034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.248017    4449 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 09:35:24.248123 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.248084    4449 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-217.us-east-2.compute.internal"
Nov 02 09:35:24.267285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.267261    4449 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 09:35:24.281657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.281636    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvstx\" (UniqueName: \"kubernetes.io/projected/cd36afed-d103-4f7a-ba99-00c841bb9537-kube-api-access-rvstx\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.281709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.281663    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:24.281709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.281680    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-systemd-units\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.281709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.281694    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-kubernetes\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.281790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.281745    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysctl-conf\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.281869 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.281856    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-lib-modules\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.281908 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.281893    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-host\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.281944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.281928    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gffwf\" (UniqueName: \"kubernetes.io/projected/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-kube-api-access-gffwf\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:35:24.281991 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.281950    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:24.282074 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282061    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovn-node-metrics-cert\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.282162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282106    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-service-ca-bundle\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:24.282162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282121    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60edc0c2-d978-40b5-b155-152019a3bdf5-config-volume\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:24.282281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282162    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-cni-multus\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.282281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282182    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysctl-d\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.282281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282248    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/69f3517a-6eb0-4b5a-a5de-fa56577626be-original-pull-secret\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:24.282383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282303    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-cni-bin\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.282383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282338    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/86739438-2924-4642-9d34-ed545e1739ba-archives-path\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:24.282436 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282402    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/86739438-2924-4642-9d34-ed545e1739ba-archives-path\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:24.282467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282439    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.282467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282459    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-netns\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.282540 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282482    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-systemd\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.282540 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282499    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-tmp-dir\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:35:24.282540 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282530    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-run-ovn-kubernetes\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.282673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282552    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-socket-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.282673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282583    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:24.282673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282594    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-tmp-dir\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:35:24.282673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282619    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.282673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282663    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bjq\" (UniqueName: \"kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq\") pod \"network-check-target-2gvpb\" (UID: \"10db22d0-6805-4d79-a670-e7b96a3b9c22\") " pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:24.282844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282659    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-service-ca-bundle\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:24.282844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282690    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovn-node-metrics-cert\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.282844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282723    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60edc0c2-d978-40b5-b155-152019a3bdf5-config-volume\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:24.282844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282751    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50bebf56-2ddc-47e0-b8f7-008d202e3af3-tmp\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.283014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282855    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5cf1aef2-0bc5-445e-880d-90215888057a-iptables-alerter-script\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:35:24.283014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282876    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.283014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282915    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50bebf56-2ddc-47e0-b8f7-008d202e3af3-tmp\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.283014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282940    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef320c69-127d-437a-87ed-cfcd9db66325-agent-certs\") pod \"konnectivity-agent-h48j7\" (UID: \"ef320c69-127d-437a-87ed-cfcd9db66325\") " pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:35:24.283014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282960    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.283014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.282990    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-conf-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.283014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283011    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-run\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.283294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283026    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:24.283294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283049    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lptr9\" (UniqueName: \"kubernetes.io/projected/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-api-access-lptr9\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:24.283294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283074    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm85w\" (UniqueName: \"kubernetes.io/projected/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-kube-api-access-wm85w\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:24.283294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283105    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.283294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283120    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5cf1aef2-0bc5-445e-880d-90215888057a-iptables-alerter-script\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:35:24.283294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283163    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef320c69-127d-437a-87ed-cfcd9db66325-agent-certs\") pod \"konnectivity-agent-h48j7\" (UID: \"ef320c69-127d-437a-87ed-cfcd9db66325\") " pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:35:24.283294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283171    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-kubelet\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.283294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283261    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef320c69-127d-437a-87ed-cfcd9db66325-konnectivity-ca\") pod \"konnectivity-agent-h48j7\" (UID: \"ef320c69-127d-437a-87ed-cfcd9db66325\") " pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:35:24.283505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283297    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbngm\" (UniqueName: \"kubernetes.io/projected/b0a20d22-7ea2-4cb8-a46f-d48043889d6f-kube-api-access-lbngm\") pod \"ingress-canary-l7v8r\" (UID: \"b0a20d22-7ea2-4cb8-a46f-d48043889d6f\") " pod="openshift-ingress-canary/ingress-canary-l7v8r"
Nov 02 09:35:24.283505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283324    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-slash\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.283505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283383    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-ovn\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.283505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283411    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-cni-netd\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.283505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283440    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60edc0c2-d978-40b5-b155-152019a3bdf5-tmp-dir\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:24.283505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283448    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef320c69-127d-437a-87ed-cfcd9db66325-konnectivity-ca\") pod \"konnectivity-agent-h48j7\" (UID: \"ef320c69-127d-437a-87ed-cfcd9db66325\") " pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:35:24.283505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283469    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwnd7\" (UniqueName: \"kubernetes.io/projected/60edc0c2-d978-40b5-b155-152019a3bdf5-kube-api-access-wwnd7\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:24.283505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283495    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5qh\" (UniqueName: \"kubernetes.io/projected/1ee8e64e-98c3-442e-9885-f45afd29a812-kube-api-access-sl5qh\") pod \"migrator-66df5d496b-ng2mz\" (UID: \"1ee8e64e-98c3-442e-9885-f45afd29a812\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz"
Nov 02 09:35:24.283747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283507    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60edc0c2-d978-40b5-b155-152019a3bdf5-tmp-dir\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:24.283747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283512    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.283747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283630    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-hostroot\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.283747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283652    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18e3a7af-aafb-4491-b696-95f4858b585a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.283747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283678    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-hosts-file\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:35:24.283747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283709    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/69f3517a-6eb0-4b5a-a5de-fa56577626be-dbus\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:24.283747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283725    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-node-log\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.283926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283774    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.283926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283804    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovnkube-script-lib\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.283926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283848    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.283926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283893    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-etc-kubernetes\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.283926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283917    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-sys\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.284060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283943    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-sys-fs\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.284060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.283982    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9sfz\" (UniqueName: \"kubernetes.io/projected/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-kube-api-access-h9sfz\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:35:24.284060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284002    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-var-lib-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.284060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284033    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.284060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284040    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovnkube-script-lib\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.284060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284056    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-device-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.284238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284087    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-cni-binary-copy\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.284238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284130    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-cni-bin\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.284238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284172    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-host\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:35:24.284238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284194    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-serviceca\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:35:24.284359 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284238    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-log-socket\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.284359 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284257    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cf1aef2-0bc5-445e-880d-90215888057a-host-slash\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:35:24.284359 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284280    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-trusted-ca-bundle\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:24.284359 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284297    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-system-cni-dir\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.284359 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284348    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-cni-binary-copy\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.284599 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284343    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-system-cni-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.284652 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284630    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:24.284701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284677    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-run-netns\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.284701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284678    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-serviceca\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:35:24.284783 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284713    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-systemd\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.284883 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284841    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-cni-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.285283 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.284975    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-var-lib-kubelet\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.285283 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285046    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c108bc48-9b97-46d1-a353-04a6dbc9ea83-data-volume\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:24.285283 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285078    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:24.285283 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285138    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovnkube-config\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.285283 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285165    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c108bc48-9b97-46d1-a353-04a6dbc9ea83-data-volume\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:24.285283 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285200    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-k8s-cni-cncf-io\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.285283 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285266    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-kubelet\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.285630 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285309    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-tuned\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.285630 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285417    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-etc-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.285630 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285448    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-ovnkube-config\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.285630 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285452    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-multus-certs\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.285630 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285533    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-tuned\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.285846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285648    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-modprobe-d\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.285846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285688    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c108bc48-9b97-46d1-a353-04a6dbc9ea83-crio-socket\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:24.285846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285718    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/69f3517a-6eb0-4b5a-a5de-fa56577626be-kubelet-config\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:24.285846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285758    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-env-overrides\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.285846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285793    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-registration-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.285846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285826    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-cnibin\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.286090 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285861    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qq5\" (UniqueName: \"kubernetes.io/projected/260f9ad0-a637-4423-ac4c-626c79e6bca9-kube-api-access-r5qq5\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.286090 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285898    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-etc-selinux\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.286090 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285935    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-cnibin\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.286090 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.285972    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqq6\" (UniqueName: \"kubernetes.io/projected/18e3a7af-aafb-4491-b696-95f4858b585a-kube-api-access-5fqq6\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.286090 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.286009    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysconfig\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.286090 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.286039    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nf7t\" (UniqueName: \"kubernetes.io/projected/50bebf56-2ddc-47e0-b8f7-008d202e3af3-kube-api-access-9nf7t\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.286090 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.286071    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0a20d22-7ea2-4cb8-a46f-d48043889d6f-cert\") pod \"ingress-canary-l7v8r\" (UID: \"b0a20d22-7ea2-4cb8-a46f-d48043889d6f\") " pod="openshift-ingress-canary/ingress-canary-l7v8r"
Nov 02 09:35:24.286403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.286108    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xbhc\" (UniqueName: \"kubernetes.io/projected/5cf1aef2-0bc5-445e-880d-90215888057a-kube-api-access-6xbhc\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:35:24.286403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.286142    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsmrc\" (UniqueName: \"kubernetes.io/projected/86739438-2924-4642-9d34-ed545e1739ba-kube-api-access-lsmrc\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:24.286403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.286178    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-socket-dir-parent\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.286403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.286205    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-daemon-config\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.286403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.286262    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttf7t\" (UniqueName: \"kubernetes.io/projected/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-kube-api-access-ttf7t\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.286601 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.286434    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-trusted-ca-bundle\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:24.286841 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.286819    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-daemon-config\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.287087 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.287068    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/260f9ad0-a637-4423-ac4c-626c79e6bca9-env-overrides\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.287392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.287374    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0a20d22-7ea2-4cb8-a46f-d48043889d6f-cert\") pod \"ingress-canary-l7v8r\" (UID: \"b0a20d22-7ea2-4cb8-a46f-d48043889d6f\") " pod="openshift-ingress-canary/ingress-canary-l7v8r"
Nov 02 09:35:24.312895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.312876    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9sfz\" (UniqueName: \"kubernetes.io/projected/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-kube-api-access-h9sfz\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:35:24.313964 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.313943    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttf7t\" (UniqueName: \"kubernetes.io/projected/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-kube-api-access-ttf7t\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.314860 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.314845    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvstx\" (UniqueName: \"kubernetes.io/projected/cd36afed-d103-4f7a-ba99-00c841bb9537-kube-api-access-rvstx\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.316005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.315989    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm85w\" (UniqueName: \"kubernetes.io/projected/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-kube-api-access-wm85w\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:24.316512 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.316492    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7bjq\" (UniqueName: \"kubernetes.io/projected/10db22d0-6805-4d79-a670-e7b96a3b9c22-kube-api-access-j7bjq\") pod \"network-check-target-2gvpb\" (UID: \"10db22d0-6805-4d79-a670-e7b96a3b9c22\") " pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:24.318050 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.318034    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5qh\" (UniqueName: \"kubernetes.io/projected/1ee8e64e-98c3-442e-9885-f45afd29a812-kube-api-access-sl5qh\") pod \"migrator-66df5d496b-ng2mz\" (UID: \"1ee8e64e-98c3-442e-9885-f45afd29a812\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ng2mz"
Nov 02 09:35:24.318425 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.318411    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gffwf\" (UniqueName: \"kubernetes.io/projected/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-kube-api-access-gffwf\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:35:24.320338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.320323    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lptr9\" (UniqueName: \"kubernetes.io/projected/c108bc48-9b97-46d1-a353-04a6dbc9ea83-kube-api-access-lptr9\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:24.324288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.324272    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qq5\" (UniqueName: \"kubernetes.io/projected/260f9ad0-a637-4423-ac4c-626c79e6bca9-kube-api-access-r5qq5\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.334757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.334737    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsmrc\" (UniqueName: \"kubernetes.io/projected/86739438-2924-4642-9d34-ed545e1739ba-kube-api-access-lsmrc\") pod \"periodic-gathering-pgbtq-2c6rz\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") " pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:35:24.339098 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.339080    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fqq6\" (UniqueName: \"kubernetes.io/projected/18e3a7af-aafb-4491-b696-95f4858b585a-kube-api-access-5fqq6\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.339633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.339612    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwnd7\" (UniqueName: \"kubernetes.io/projected/60edc0c2-d978-40b5-b155-152019a3bdf5-kube-api-access-wwnd7\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:24.340823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.340804    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xbhc\" (UniqueName: \"kubernetes.io/projected/5cf1aef2-0bc5-445e-880d-90215888057a-kube-api-access-6xbhc\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:35:24.341508 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.341492    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nf7t\" (UniqueName: \"kubernetes.io/projected/50bebf56-2ddc-47e0-b8f7-008d202e3af3-kube-api-access-9nf7t\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.341925 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.341910    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbngm\" (UniqueName: \"kubernetes.io/projected/b0a20d22-7ea2-4cb8-a46f-d48043889d6f-kube-api-access-lbngm\") pod \"ingress-canary-l7v8r\" (UID: \"b0a20d22-7ea2-4cb8-a46f-d48043889d6f\") " pod="openshift-ingress-canary/ingress-canary-l7v8r"
Nov 02 09:35:24.386753 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.386734    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysconfig\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.386845 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.386761    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-socket-dir-parent\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.386845 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.386777    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:24.386845 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.386827    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-systemd-units\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.386845 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.386845    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-kubernetes\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.386997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.386851    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysconfig\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.386997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.386872    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-socket-dir-parent\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.386997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.386886    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-kubernetes\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.386997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.386891    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-systemd-units\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.386997 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.386907    4449 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 09:35:24.386997 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.386988    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs podName:d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.886954959 +0000 UTC m=+1.825241739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs") pod "network-metrics-daemon-p8bj8" (UID: "d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa") : secret "metrics-daemon-secret" not found
Nov 02 09:35:24.387204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387003    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysctl-conf\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.387204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387021    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-lib-modules\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.387204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387040    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-host\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.387204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387065    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-cni-multus\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.387204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387085    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysctl-d\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.387204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387095    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-host\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.387204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387103    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-cni-bin\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.387204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387139    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-cni-multus\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.387204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387143    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.387204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387143    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-lib-modules\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.387204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387140    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysctl-conf\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.387204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387196    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-netns\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.387204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387204    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-cni-bin\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387261    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-systemd\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387258    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387273    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-netns\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387279    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-sysctl-d\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387310    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-run-ovn-kubernetes\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387332    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-socket-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387346    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387363    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-systemd\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387387    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-conf-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.387406    4449 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387411    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-run\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387426    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387437    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-conf-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387412    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-socket-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387451    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-run\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.387460    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls podName:60edc0c2-d978-40b5-b155-152019a3bdf5 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.887445072 +0000 UTC m=+1.825731840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls") pod "dns-default-sjj6v" (UID: "60edc0c2-d978-40b5-b155-152019a3bdf5") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387403    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-run-ovn-kubernetes\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387445    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-kubelet\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387474    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-kubelet\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387496    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-slash\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.387513    4449 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387523    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-ovn\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387537    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-cni-netd\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387544    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-slash\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.387554    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls podName:c108bc48-9b97-46d1-a353-04a6dbc9ea83 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.887544238 +0000 UTC m=+1.825830991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smlj2" (UID: "c108bc48-9b97-46d1-a353-04a6dbc9ea83") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387568    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-cni-netd\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387575    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-hostroot\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387582    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-ovn\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387590    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-hosts-file\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387609    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/69f3517a-6eb0-4b5a-a5de-fa56577626be-dbus\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387620    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8e9ee3cd-e542-40a1-a18f-2843dcd82a69-hosts-file\") pod \"node-resolver-r85rg\" (UID: \"8e9ee3cd-e542-40a1-a18f-2843dcd82a69\") " pod="openshift-dns/node-resolver-r85rg"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387624    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-node-log\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387642    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-hostroot\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.387648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387663    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387677    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-node-log\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387694    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387711    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/69f3517a-6eb0-4b5a-a5de-fa56577626be-dbus\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387730    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387750    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387758    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-etc-kubernetes\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387778    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-sys\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387792    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-sys-fs\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387807    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-var-lib-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387815    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-sys\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387792    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-etc-kubernetes\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387845    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387871    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-device-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387850    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-sys-fs\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387894    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-cni-bin\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387913    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387916    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-host\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387927    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-var-lib-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387932    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-device-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387945    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-log-socket\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387951    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0556aaf2-dc7e-4e57-a721-4e0ee6cfd923-host\") pod \"node-ca-hxwln\" (UID: \"0556aaf2-dc7e-4e57-a721-4e0ee6cfd923\") " pod="openshift-image-registry/node-ca-hxwln"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387972    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cf1aef2-0bc5-445e-880d-90215888057a-host-slash\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387950    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-cni-bin\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.387992    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-log-socket\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388023    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-system-cni-dir\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388033    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cf1aef2-0bc5-445e-880d-90215888057a-host-slash\") pod \"iptables-alerter-r8j97\" (UID: \"5cf1aef2-0bc5-445e-880d-90215888057a\") " pod="openshift-network-operator/iptables-alerter-r8j97"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388071    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-system-cni-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388074    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-system-cni-dir\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388122    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-run-netns\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388142    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-systemd\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388158    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-cni-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388191    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-system-cni-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388257    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-host-run-netns\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388260    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-run-systemd\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388281    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-var-lib-kubelet\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388295    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-multus-cni-dir\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388315    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-k8s-cni-cncf-io\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388339    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-kubelet\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388362    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-k8s-cni-cncf-io\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388364    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-etc-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388377    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-var-lib-kubelet\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388386    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-var-lib-kubelet\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388391    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-multus-certs\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388422    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-host-run-multus-certs\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388422    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/260f9ad0-a637-4423-ac4c-626c79e6bca9-etc-openvswitch\") pod \"ovnkube-node-px4ck\" (UID: \"260f9ad0-a637-4423-ac4c-626c79e6bca9\") " pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388426    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-modprobe-d\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388451    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c108bc48-9b97-46d1-a353-04a6dbc9ea83-crio-socket\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388465    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/69f3517a-6eb0-4b5a-a5de-fa56577626be-kubelet-config\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388480    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-registration-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388516    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-cnibin\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388526    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c108bc48-9b97-46d1-a353-04a6dbc9ea83-crio-socket\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388515    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50bebf56-2ddc-47e0-b8f7-008d202e3af3-etc-modprobe-d\") pod \"tuned-jxnjv\" (UID: \"50bebf56-2ddc-47e0-b8f7-008d202e3af3\") " pod="openshift-cluster-node-tuning-operator/tuned-jxnjv"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388541    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/69f3517a-6eb0-4b5a-a5de-fa56577626be-kubelet-config\") pod \"global-pull-secret-syncer-8w4cb\" (UID: \"69f3517a-6eb0-4b5a-a5de-fa56577626be\") " pod="kube-system/global-pull-secret-syncer-8w4cb"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388560    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-registration-dir\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388585    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fab1d8c2-b3fa-43bc-9009-a0a9c3079eab-cnibin\") pod \"multus-vkr2t\" (UID: \"fab1d8c2-b3fa-43bc-9009-a0a9c3079eab\") " pod="openshift-multus/multus-vkr2t"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388533    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-etc-selinux\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388613    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-cnibin\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388590    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd36afed-d103-4f7a-ba99-00c841bb9537-etc-selinux\") pod \"aws-ebs-csi-driver-node-bv9sd\" (UID: \"cd36afed-d103-4f7a-ba99-00c841bb9537\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bv9sd"
Nov 02 09:35:24.388926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.388649    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18e3a7af-aafb-4491-b696-95f4858b585a-cnibin\") pod \"multus-additional-cni-plugins-8pbvj\" (UID: \"18e3a7af-aafb-4491-b696-95f4858b585a\") " pod="openshift-multus/multus-additional-cni-plugins-8pbvj"
Nov 02 09:35:24.892652 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.892624    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:24.892771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.892663    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:24.892771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.892682    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:24.892844 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.892772    4449 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:24.892844 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.892774    4449 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:24.892844 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.892774    4449 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 09:35:24.892844 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.892824    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls podName:c108bc48-9b97-46d1-a353-04a6dbc9ea83 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:25.892811386 +0000 UTC m=+2.831098140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smlj2" (UID: "c108bc48-9b97-46d1-a353-04a6dbc9ea83") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:24.892952 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.892853    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls podName:60edc0c2-d978-40b5-b155-152019a3bdf5 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:25.892847146 +0000 UTC m=+2.831133901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls") pod "dns-default-sjj6v" (UID: "60edc0c2-d978-40b5-b155-152019a3bdf5") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:24.892952 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:24.892862    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs podName:d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa nodeName:}" failed. No retries permitted until 2026-11-02 09:35:25.892857288 +0000 UTC m=+2.831144041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs") pod "network-metrics-daemon-p8bj8" (UID: "d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa") : secret "metrics-daemon-secret" not found
Nov 02 09:35:24.945912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:24.945855    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l7v8r_b0a20d22-7ea2-4cb8-a46f-d48043889d6f/serve-healthcheck-canary/0.log"
Nov 02 09:35:25.546425 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:25.546402    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ng2mz_1ee8e64e-98c3-442e-9885-f45afd29a812/migrator/0.log"
Nov 02 09:35:25.898084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:25.898018    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:25.898084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:25.898054    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:25.898084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:25.898073    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:25.898368 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:25.898166    4449 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:25.898368 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:25.898171    4449 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 09:35:25.898368 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:25.898176    4449 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:25.898368 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:25.898240    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls podName:c108bc48-9b97-46d1-a353-04a6dbc9ea83 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:27.898204491 +0000 UTC m=+4.836491247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smlj2" (UID: "c108bc48-9b97-46d1-a353-04a6dbc9ea83") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:25.898368 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:25.898253    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls podName:60edc0c2-d978-40b5-b155-152019a3bdf5 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:27.898247108 +0000 UTC m=+4.836533863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls") pod "dns-default-sjj6v" (UID: "60edc0c2-d978-40b5-b155-152019a3bdf5") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:25.898368 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:25.898265    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs podName:d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa nodeName:}" failed. No retries permitted until 2026-11-02 09:35:27.898257333 +0000 UTC m=+4.836544107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs") pod "network-metrics-daemon-p8bj8" (UID: "d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa") : secret "metrics-daemon-secret" not found
Nov 02 09:35:26.483568 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:26.483528    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:27.911931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:27.911890    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:27.911931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:27.911936    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:27.912457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:27.911959    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:27.912457 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:27.912049    4449 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:27.912457 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:27.912055    4449 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:27.912457 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:27.912106    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls podName:c108bc48-9b97-46d1-a353-04a6dbc9ea83 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:31.912093166 +0000 UTC m=+8.850379920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smlj2" (UID: "c108bc48-9b97-46d1-a353-04a6dbc9ea83") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:27.912457 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:27.912056    4449 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 09:35:27.912457 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:27.912176    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls podName:60edc0c2-d978-40b5-b155-152019a3bdf5 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:31.9121612 +0000 UTC m=+8.850447974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls") pod "dns-default-sjj6v" (UID: "60edc0c2-d978-40b5-b155-152019a3bdf5") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:27.912457 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:27.912243    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs podName:d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa nodeName:}" failed. No retries permitted until 2026-11-02 09:35:31.91220454 +0000 UTC m=+8.850491308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs") pod "network-metrics-daemon-p8bj8" (UID: "d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa") : secret "metrics-daemon-secret" not found
Nov 02 09:35:28.462374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:28.462346    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:29.312819 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:29.312789    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:29.313184 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:29.312961    4449 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 09:35:29.778615 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:29.778546    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-px4ck"
Nov 02 09:35:31.939421 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:31.939387    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:31.939841 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:31.939430    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:31.939841 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:31.939452    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:31.939841 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:31.939545    4449 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:31.939841 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:31.939625    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls podName:60edc0c2-d978-40b5-b155-152019a3bdf5 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:39.939609458 +0000 UTC m=+16.877896232 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls") pod "dns-default-sjj6v" (UID: "60edc0c2-d978-40b5-b155-152019a3bdf5") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:31.939841 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:31.939802    4449 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 09:35:31.942034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:31.942003    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c108bc48-9b97-46d1-a353-04a6dbc9ea83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smlj2\" (UID: \"c108bc48-9b97-46d1-a353-04a6dbc9ea83\") " pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:31.942034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:31.942010    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa-metrics-certs\") pod \"network-metrics-daemon-p8bj8\" (UID: \"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa\") " pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:31.980069 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:31.980032    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8bj8"
Nov 02 09:35:31.985948 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:31.985926    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-smlj2"
Nov 02 09:35:32.252756 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:32.252674    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8bj8"]
Nov 02 09:35:32.256253 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:32.256208    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e5827f_c33d_4b1a_b2ba_16ac6ec07efa.slice/crio-01ef44d8e7f6a1db92d78eb9826f2f5e93c95bfb0f5148ae545402c24da691ba WatchSource:0}: Error finding container 01ef44d8e7f6a1db92d78eb9826f2f5e93c95bfb0f5148ae545402c24da691ba: Status 404 returned error can't find the container with id 01ef44d8e7f6a1db92d78eb9826f2f5e93c95bfb0f5148ae545402c24da691ba
Nov 02 09:35:32.259127 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:32.258202    4449 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 09:35:32.259127 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:32.259035    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-smlj2"]
Nov 02 09:35:32.263739 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:32.263715    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc108bc48_9b97_46d1_a353_04a6dbc9ea83.slice/crio-e5aeae7cc808789018a7cfda847cc67cb248036051c2874cf1b93dc2e0dd1bc2 WatchSource:0}: Error finding container e5aeae7cc808789018a7cfda847cc67cb248036051c2874cf1b93dc2e0dd1bc2: Status 404 returned error can't find the container with id e5aeae7cc808789018a7cfda847cc67cb248036051c2874cf1b93dc2e0dd1bc2
Nov 02 09:35:33.261247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:33.261187    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-smlj2" event={"ID":"c108bc48-9b97-46d1-a353-04a6dbc9ea83","Type":"ContainerStarted","Data":"f39f45f221f02f4eb5f29046a2f400ae65cf9195197dfeef0555ba3f38d6e483"}
Nov 02 09:35:33.261660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:33.261270    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-smlj2" event={"ID":"c108bc48-9b97-46d1-a353-04a6dbc9ea83","Type":"ContainerStarted","Data":"f7290a36154f34866ea0efd6407710210c3f7264663bfb432a345356e42cb772"}
Nov 02 09:35:33.261660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:33.261302    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-smlj2" event={"ID":"c108bc48-9b97-46d1-a353-04a6dbc9ea83","Type":"ContainerStarted","Data":"e5aeae7cc808789018a7cfda847cc67cb248036051c2874cf1b93dc2e0dd1bc2"}
Nov 02 09:35:33.262664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:33.262640    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8bj8" event={"ID":"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa","Type":"ContainerStarted","Data":"01ef44d8e7f6a1db92d78eb9826f2f5e93c95bfb0f5148ae545402c24da691ba"}
Nov 02 09:35:34.267082 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:34.267049    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8bj8" event={"ID":"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa","Type":"ContainerStarted","Data":"5df7f50287f129cf8facda1da39c38078890ad3509d8e24ef9b2e13a885290e7"}
Nov 02 09:35:34.267485 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:34.267123    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8bj8" event={"ID":"d0e5827f-c33d-4b1a-b2ba-16ac6ec07efa","Type":"ContainerStarted","Data":"0764942ba928af465fb249b18c600820d182e2157562a7a600b265c5bef13d76"}
Nov 02 09:35:35.274549 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:35.274441    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-smlj2" event={"ID":"c108bc48-9b97-46d1-a353-04a6dbc9ea83","Type":"ContainerStarted","Data":"6f4c987ac5f70b59ae13c59deb7330e4a43819d24d3d13474daface6574a06d9"}
Nov 02 09:35:36.005092 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:36.005062    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h48j7"
Nov 02 09:35:36.544832 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:35:36.544801    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:35:37.816367 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.816333    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84b96dbbff-vb64t"]
Nov 02 09:35:37.819731 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.819702    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.829799 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.829779    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 09:35:37.833995 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.833971    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84b96dbbff-vb64t"]
Nov 02 09:35:37.837916 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.837896    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 09:35:37.838448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.838396    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 09:35:37.838448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.838408    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 09:35:37.838448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.838415    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cmxq6\""
Nov 02 09:35:37.889578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.889547    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67fc47bc-862f-4311-918f-2af036ebd4b4-bound-sa-token\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.889578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.889583    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67fc47bc-862f-4311-918f-2af036ebd4b4-image-registry-private-configuration\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.889760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.889600    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67fc47bc-862f-4311-918f-2af036ebd4b4-ca-trust-extracted\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.889760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.889619    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67fc47bc-862f-4311-918f-2af036ebd4b4-registry-certificates\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.889760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.889676    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llrvg\" (UniqueName: \"kubernetes.io/projected/67fc47bc-862f-4311-918f-2af036ebd4b4-kube-api-access-llrvg\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.889760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.889743    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67fc47bc-862f-4311-918f-2af036ebd4b4-trusted-ca\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.889884 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.889772    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67fc47bc-862f-4311-918f-2af036ebd4b4-registry-tls\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.889884 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.889860    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67fc47bc-862f-4311-918f-2af036ebd4b4-installation-pull-secrets\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.991080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.991047    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67fc47bc-862f-4311-918f-2af036ebd4b4-registry-tls\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.991243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.991098    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67fc47bc-862f-4311-918f-2af036ebd4b4-installation-pull-secrets\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.991243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.991122    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67fc47bc-862f-4311-918f-2af036ebd4b4-bound-sa-token\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.991243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.991142    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67fc47bc-862f-4311-918f-2af036ebd4b4-image-registry-private-configuration\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.991243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.991158    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67fc47bc-862f-4311-918f-2af036ebd4b4-ca-trust-extracted\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.991243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.991178    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67fc47bc-862f-4311-918f-2af036ebd4b4-registry-certificates\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.991243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.991201    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llrvg\" (UniqueName: \"kubernetes.io/projected/67fc47bc-862f-4311-918f-2af036ebd4b4-kube-api-access-llrvg\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.991537 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.991258    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67fc47bc-862f-4311-918f-2af036ebd4b4-trusted-ca\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.991783 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.991758    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67fc47bc-862f-4311-918f-2af036ebd4b4-ca-trust-extracted\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.992335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.992315    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67fc47bc-862f-4311-918f-2af036ebd4b4-trusted-ca\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.992559 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.992532    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67fc47bc-862f-4311-918f-2af036ebd4b4-registry-certificates\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.993760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.993736    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67fc47bc-862f-4311-918f-2af036ebd4b4-image-registry-private-configuration\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.993880 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.993861    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67fc47bc-862f-4311-918f-2af036ebd4b4-installation-pull-secrets\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:37.993940 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:37.993920    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67fc47bc-862f-4311-918f-2af036ebd4b4-registry-tls\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:38.003233 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:38.003192    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67fc47bc-862f-4311-918f-2af036ebd4b4-bound-sa-token\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:38.003728 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:38.003711    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llrvg\" (UniqueName: \"kubernetes.io/projected/67fc47bc-862f-4311-918f-2af036ebd4b4-kube-api-access-llrvg\") pod \"image-registry-84b96dbbff-vb64t\" (UID: \"67fc47bc-862f-4311-918f-2af036ebd4b4\") " pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:38.130957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:38.130880    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:38.406438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:38.406359    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84b96dbbff-vb64t"]
Nov 02 09:35:38.410370 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:38.410333    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fc47bc_862f_4311_918f_2af036ebd4b4.slice/crio-6a7465fc956dec8c117ce35fe9a11228c38a83ed3ce94df3d7e39599ca5a7512 WatchSource:0}: Error finding container 6a7465fc956dec8c117ce35fe9a11228c38a83ed3ce94df3d7e39599ca5a7512: Status 404 returned error can't find the container with id 6a7465fc956dec8c117ce35fe9a11228c38a83ed3ce94df3d7e39599ca5a7512
Nov 02 09:35:39.287249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:39.287200    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84b96dbbff-vb64t" event={"ID":"67fc47bc-862f-4311-918f-2af036ebd4b4","Type":"ContainerStarted","Data":"718bdce161247c45dee0a52bac81d585d26258f09b317b4bf5a4da4b2c5b85c0"}
Nov 02 09:35:39.287620 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:39.287266    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84b96dbbff-vb64t" event={"ID":"67fc47bc-862f-4311-918f-2af036ebd4b4","Type":"ContainerStarted","Data":"6a7465fc956dec8c117ce35fe9a11228c38a83ed3ce94df3d7e39599ca5a7512"}
Nov 02 09:35:39.287620 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:39.287324    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:35:39.311989 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:39.311775    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84b96dbbff-vb64t" podStartSLOduration=2.311756854 podStartE2EDuration="2.311756854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:39.310162252 +0000 UTC m=+16.248449089" watchObservedRunningTime="2026-11-02 09:35:39.311756854 +0000 UTC m=+16.250043628"
Nov 02 09:35:40.006007 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:40.005961    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:40.008389 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:40.008363    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60edc0c2-d978-40b5-b155-152019a3bdf5-metrics-tls\") pod \"dns-default-sjj6v\" (UID: \"60edc0c2-d978-40b5-b155-152019a3bdf5\") " pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:40.083149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:40.083113    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:40.353577 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:40.353545    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sjj6v"]
Nov 02 09:35:40.356735 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:40.356709    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60edc0c2_d978_40b5_b155_152019a3bdf5.slice/crio-da0dd8549ccd807b401468501bcf8a194d1242918f19e37a4b9e9dae14fe34d7 WatchSource:0}: Error finding container da0dd8549ccd807b401468501bcf8a194d1242918f19e37a4b9e9dae14fe34d7: Status 404 returned error can't find the container with id da0dd8549ccd807b401468501bcf8a194d1242918f19e37a4b9e9dae14fe34d7
Nov 02 09:35:41.298197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:41.298151    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sjj6v" event={"ID":"60edc0c2-d978-40b5-b155-152019a3bdf5","Type":"ContainerStarted","Data":"da0dd8549ccd807b401468501bcf8a194d1242918f19e37a4b9e9dae14fe34d7"}
Nov 02 09:35:43.306099 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:43.306064    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sjj6v" event={"ID":"60edc0c2-d978-40b5-b155-152019a3bdf5","Type":"ContainerStarted","Data":"8de7d9474a25c29445570fac604e29fd94f6388d74a2a5dbd0ac808715f7971b"}
Nov 02 09:35:43.306548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:43.306115    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sjj6v" event={"ID":"60edc0c2-d978-40b5-b155-152019a3bdf5","Type":"ContainerStarted","Data":"066a6bd604bece2cb7ff8bc99f8503c3248b4aac3c41400b335934519b23a4cc"}
Nov 02 09:35:43.306548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:43.306238    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:44.249103 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.249065    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"]
Nov 02 09:35:44.252127 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.252109    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.264157 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.264135    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 09:35:44.264325 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.264310    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-622sp\""
Nov 02 09:35:44.264455 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.264440    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 09:35:44.264655 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.264643    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 09:35:44.264709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.264682    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 09:35:44.266680 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.266648    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"]
Nov 02 09:35:44.267025 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.267005    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 09:35:44.278275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.278240    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8cgjf"]
Nov 02 09:35:44.281050 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.281022    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.338087 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.338056    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc4zm\" (UniqueName: \"kubernetes.io/projected/e7929f42-1f6b-4ad9-866b-da15457ab4ec-kube-api-access-xc4zm\") pod \"openshift-state-metrics-7bd7489957-fvz6m\" (UID: \"e7929f42-1f6b-4ad9-866b-da15457ab4ec\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.338087 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.338090    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.338605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.338110    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.338605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.338180    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl9nx\" (UniqueName: \"kubernetes.io/projected/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-kube-api-access-kl9nx\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.338605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.338323    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-sys\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.338605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.338361    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7929f42-1f6b-4ad9-866b-da15457ab4ec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fvz6m\" (UID: \"e7929f42-1f6b-4ad9-866b-da15457ab4ec\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.338605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.338459    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-textfile\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.338605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.338478    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-tls\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.338605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.338506    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-root\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.338605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.338528    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7929f42-1f6b-4ad9-866b-da15457ab4ec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fvz6m\" (UID: \"e7929f42-1f6b-4ad9-866b-da15457ab4ec\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.338605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.338548    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7929f42-1f6b-4ad9-866b-da15457ab4ec-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fvz6m\" (UID: \"e7929f42-1f6b-4ad9-866b-da15457ab4ec\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.338605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.338570    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-wtmp\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.338919 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.338668    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-metrics-client-ca\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.341091 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.341065    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 09:35:44.341242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.341107    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 09:35:44.341322 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.341303    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-phbth\""
Nov 02 09:35:44.361813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.361789    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 09:35:44.439929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.439892    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-metrics-client-ca\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.439929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.439931    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc4zm\" (UniqueName: \"kubernetes.io/projected/e7929f42-1f6b-4ad9-866b-da15457ab4ec-kube-api-access-xc4zm\") pod \"openshift-state-metrics-7bd7489957-fvz6m\" (UID: \"e7929f42-1f6b-4ad9-866b-da15457ab4ec\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.440168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.439963    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.440168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.439981    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.440168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440021    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl9nx\" (UniqueName: \"kubernetes.io/projected/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-kube-api-access-kl9nx\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.440168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440049    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-sys\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.440168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440077    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7929f42-1f6b-4ad9-866b-da15457ab4ec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fvz6m\" (UID: \"e7929f42-1f6b-4ad9-866b-da15457ab4ec\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.440168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440131    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-textfile\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.440168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440151    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-tls\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.440550 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440175    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-root\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.440550 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440202    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7929f42-1f6b-4ad9-866b-da15457ab4ec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fvz6m\" (UID: \"e7929f42-1f6b-4ad9-866b-da15457ab4ec\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.440550 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440254    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7929f42-1f6b-4ad9-866b-da15457ab4ec-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fvz6m\" (UID: \"e7929f42-1f6b-4ad9-866b-da15457ab4ec\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.440550 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440440    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-root\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.440550 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440545    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-textfile\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.440796 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440732    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-metrics-client-ca\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.440860 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440827    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.441014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440991    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7929f42-1f6b-4ad9-866b-da15457ab4ec-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fvz6m\" (UID: \"e7929f42-1f6b-4ad9-866b-da15457ab4ec\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.441295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.441038    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-wtmp\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.441295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.440842    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-sys\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.441410 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.441397    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-wtmp\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.442689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.442664    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.442883 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.442861    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-node-exporter-tls\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.443082 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.443058    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7929f42-1f6b-4ad9-866b-da15457ab4ec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fvz6m\" (UID: \"e7929f42-1f6b-4ad9-866b-da15457ab4ec\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.443148 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.443130    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7929f42-1f6b-4ad9-866b-da15457ab4ec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fvz6m\" (UID: \"e7929f42-1f6b-4ad9-866b-da15457ab4ec\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.452524 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.452496    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl9nx\" (UniqueName: \"kubernetes.io/projected/4dce88eb-70b7-4d91-bd14-533e9c3c18cf-kube-api-access-kl9nx\") pod \"node-exporter-8cgjf\" (UID: \"4dce88eb-70b7-4d91-bd14-533e9c3c18cf\") " pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.452616 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.452600    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc4zm\" (UniqueName: \"kubernetes.io/projected/e7929f42-1f6b-4ad9-866b-da15457ab4ec-kube-api-access-xc4zm\") pod \"openshift-state-metrics-7bd7489957-fvz6m\" (UID: \"e7929f42-1f6b-4ad9-866b-da15457ab4ec\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.568584 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.568550    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"
Nov 02 09:35:44.596698 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.596671    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8cgjf"
Nov 02 09:35:44.607747 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:44.607719    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dce88eb_70b7_4d91_bd14_533e9c3c18cf.slice/crio-a96622f0681a3b3bacf2f5b40c29c454030f51547d04e566b25ec509875eac01 WatchSource:0}: Error finding container a96622f0681a3b3bacf2f5b40c29c454030f51547d04e566b25ec509875eac01: Status 404 returned error can't find the container with id a96622f0681a3b3bacf2f5b40c29c454030f51547d04e566b25ec509875eac01
Nov 02 09:35:44.844310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:44.844194    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m"]
Nov 02 09:35:44.847854 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:44.847810    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7929f42_1f6b_4ad9_866b_da15457ab4ec.slice/crio-7f83b86ed63369a9d638b4b803d3542684d65e19a9c5be92bbe98aaa25ad4b86 WatchSource:0}: Error finding container 7f83b86ed63369a9d638b4b803d3542684d65e19a9c5be92bbe98aaa25ad4b86: Status 404 returned error can't find the container with id 7f83b86ed63369a9d638b4b803d3542684d65e19a9c5be92bbe98aaa25ad4b86
Nov 02 09:35:45.312609 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:45.312578    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8cgjf" event={"ID":"4dce88eb-70b7-4d91-bd14-533e9c3c18cf","Type":"ContainerStarted","Data":"a96622f0681a3b3bacf2f5b40c29c454030f51547d04e566b25ec509875eac01"}
Nov 02 09:35:45.314462 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:45.314433    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m" event={"ID":"e7929f42-1f6b-4ad9-866b-da15457ab4ec","Type":"ContainerStarted","Data":"b5adda6bcfe90afb05c5c1fd84fe92105028625ac74d8807f5d635bda47344ea"}
Nov 02 09:35:45.314574 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:45.314487    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m" event={"ID":"e7929f42-1f6b-4ad9-866b-da15457ab4ec","Type":"ContainerStarted","Data":"79cb77436d12f291e90f564f18fdc530c82cb1959df1f923f999c480f90d4e7a"}
Nov 02 09:35:45.314574 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:45.314505    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m" event={"ID":"e7929f42-1f6b-4ad9-866b-da15457ab4ec","Type":"ContainerStarted","Data":"7f83b86ed63369a9d638b4b803d3542684d65e19a9c5be92bbe98aaa25ad4b86"}
Nov 02 09:35:46.319482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.319440    4449 generic.go:356] "Generic (PLEG): container finished" podID="4dce88eb-70b7-4d91-bd14-533e9c3c18cf" containerID="dc05cbfcfb4af9ccdbb1ab0ea195abee79c247e6967a829619927ece8d51dd87" exitCode=0
Nov 02 09:35:46.319867 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.319538    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8cgjf" event={"ID":"4dce88eb-70b7-4d91-bd14-533e9c3c18cf","Type":"ContainerDied","Data":"dc05cbfcfb4af9ccdbb1ab0ea195abee79c247e6967a829619927ece8d51dd87"}
Nov 02 09:35:46.321707 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.321671    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m" event={"ID":"e7929f42-1f6b-4ad9-866b-da15457ab4ec","Type":"ContainerStarted","Data":"36534eafc3b34f428dba6882c9ae4b6c88d075a81ff9cd57bf3ae05655af60fa"}
Nov 02 09:35:46.346607 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.346583    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"]
Nov 02 09:35:46.351201 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.351185    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.355578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.355558    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 09:35:46.355684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.355574    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 09:35:46.355684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.355574    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 09:35:46.355778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.355696    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cco6jj90tsu8h\""
Nov 02 09:35:46.355778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.355714    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 09:35:46.355778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.355761    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 09:35:46.355778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.355761    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j6t57\""
Nov 02 09:35:46.362548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.362504    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fvz6m" podStartSLOduration=1.309539391 podStartE2EDuration="2.362489706s" totalImagesPullingTime="1.052950315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:46.362164042 +0000 UTC m=+23.300450836" watchObservedRunningTime="2026-11-02 09:35:46.362489706 +0000 UTC m=+23.300776483"
Nov 02 09:35:46.365940 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.365915    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"]
Nov 02 09:35:46.456966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.456939    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.457191 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.457167    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f39265c-2507-4c75-969c-33cdaea8f677-tmp\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.457297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.457277    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.457358 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.457318    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.457427 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.457398    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtx2v\" (UniqueName: \"kubernetes.io/projected/5f39265c-2507-4c75-969c-33cdaea8f677-kube-api-access-jtx2v\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.457476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.457432    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.457476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.457449    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-grpc-tls\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.457586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.457573    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f39265c-2507-4c75-969c-33cdaea8f677-metrics-client-ca\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.457691 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.457670    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-tls\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.558919 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.558889    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f39265c-2507-4c75-969c-33cdaea8f677-metrics-client-ca\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.558919 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.558923    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-tls\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.559165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.558958    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.559165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.558978    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f39265c-2507-4c75-969c-33cdaea8f677-tmp\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.559165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.559009    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.559165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.559040    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.559165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.559071    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtx2v\" (UniqueName: \"kubernetes.io/projected/5f39265c-2507-4c75-969c-33cdaea8f677-kube-api-access-jtx2v\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.559165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.559099    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.559498 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.559449    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f39265c-2507-4c75-969c-33cdaea8f677-tmp\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.559498 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.559473    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-grpc-tls\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.559963 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.559841    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f39265c-2507-4c75-969c-33cdaea8f677-metrics-client-ca\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.561935 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.561905    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.562030 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.561905    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.562354 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.562332    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.562452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.562435    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.562488 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.562436    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-thanos-querier-tls\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.562659 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.562636    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f39265c-2507-4c75-969c-33cdaea8f677-secret-grpc-tls\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.573674 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.573625    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtx2v\" (UniqueName: \"kubernetes.io/projected/5f39265c-2507-4c75-969c-33cdaea8f677-kube-api-access-jtx2v\") pod \"thanos-querier-6d9cc5b7-z88nh\" (UID: \"5f39265c-2507-4c75-969c-33cdaea8f677\") " pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:46.691138 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:46.691098    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:47.005019 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:47.004933    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"]
Nov 02 09:35:47.008859 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:47.008825    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f39265c_2507_4c75_969c_33cdaea8f677.slice/crio-5f4335b02f3e32c602a38c36cf7f1af22411b9f17bb11247f3c4f599703d6126 WatchSource:0}: Error finding container 5f4335b02f3e32c602a38c36cf7f1af22411b9f17bb11247f3c4f599703d6126: Status 404 returned error can't find the container with id 5f4335b02f3e32c602a38c36cf7f1af22411b9f17bb11247f3c4f599703d6126
Nov 02 09:35:47.327022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:47.326979    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8cgjf" event={"ID":"4dce88eb-70b7-4d91-bd14-533e9c3c18cf","Type":"ContainerStarted","Data":"73e6cc9af00407007685e3c69e5639aba1ef98288b1439b30094f1ef7bfb7fa5"}
Nov 02 09:35:47.327476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:47.327045    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8cgjf" event={"ID":"4dce88eb-70b7-4d91-bd14-533e9c3c18cf","Type":"ContainerStarted","Data":"81b3ddee269a9f886e53441e9f41d27cd6bff3fd7191f46f7404163976d149a7"}
Nov 02 09:35:47.328305 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:47.328280    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh" event={"ID":"5f39265c-2507-4c75-969c-33cdaea8f677","Type":"ContainerStarted","Data":"5f4335b02f3e32c602a38c36cf7f1af22411b9f17bb11247f3c4f599703d6126"}
Nov 02 09:35:47.349856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:47.349810    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8cgjf" podStartSLOduration=1.882111425 podStartE2EDuration="3.349797677s" totalImagesPullingTime="1.391381275s" totalInitContainerRuntime="76.304977ms" isStatefulPod=true podCreationTimestamp="2026-11-02 09:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:47.348795808 +0000 UTC m=+24.287082595" watchObservedRunningTime="2026-11-02 09:35:47.349797677 +0000 UTC m=+24.288084452"
Nov 02 09:35:49.336490 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.336461    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh" event={"ID":"5f39265c-2507-4c75-969c-33cdaea8f677","Type":"ContainerStarted","Data":"f884312e82ba0b3f38d6f9a66e9beedce39ea5425e0d837410c47049612fe696"}
Nov 02 09:35:49.336847 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.336525    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh" event={"ID":"5f39265c-2507-4c75-969c-33cdaea8f677","Type":"ContainerStarted","Data":"35f7ff22ff0a00b1f8218a34dcab95ff0c9a515dfa9d20f495c12da5f45964b5"}
Nov 02 09:35:49.336847 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.336550    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh" event={"ID":"5f39265c-2507-4c75-969c-33cdaea8f677","Type":"ContainerStarted","Data":"f79470062e9b4bd3a9534b3ba81211d34bc96bd72f1b0efa8b580ccd107d780a"}
Nov 02 09:35:49.472732 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.472699    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"]
Nov 02 09:35:49.476070 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.476054    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.480294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.480265    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 09:35:49.480408 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.480386    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 09:35:49.481087 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.481070    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 09:35:49.481379 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.481358    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-p9cpw\""
Nov 02 09:35:49.481469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.481417    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 09:35:49.489178 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.489154    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"]
Nov 02 09:35:49.503412 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.503389    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 09:35:49.506588 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.506565    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 09:35:49.586298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.586263    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d915c045-08fd-4ca6-a35d-0fa360cee77c-secret-telemeter-client\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.586522 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.586321    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d915c045-08fd-4ca6-a35d-0fa360cee77c-metrics-client-ca\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.586522 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.586399    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d915c045-08fd-4ca6-a35d-0fa360cee77c-telemeter-client-tls\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.586522 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.586448    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5zfc\" (UniqueName: \"kubernetes.io/projected/d915c045-08fd-4ca6-a35d-0fa360cee77c-kube-api-access-b5zfc\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.586522 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.586515    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d915c045-08fd-4ca6-a35d-0fa360cee77c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.586689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.586536    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d915c045-08fd-4ca6-a35d-0fa360cee77c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.586689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.586564    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d915c045-08fd-4ca6-a35d-0fa360cee77c-serving-certs-ca-bundle\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.586689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.586622    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d915c045-08fd-4ca6-a35d-0fa360cee77c-federate-client-tls\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.687528 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.687488    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d915c045-08fd-4ca6-a35d-0fa360cee77c-serving-certs-ca-bundle\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.687528 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.687531    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d915c045-08fd-4ca6-a35d-0fa360cee77c-federate-client-tls\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.687711 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.687575    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d915c045-08fd-4ca6-a35d-0fa360cee77c-secret-telemeter-client\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.687711 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.687606    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d915c045-08fd-4ca6-a35d-0fa360cee77c-metrics-client-ca\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.687711 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.687634    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d915c045-08fd-4ca6-a35d-0fa360cee77c-telemeter-client-tls\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.687711 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.687656    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5zfc\" (UniqueName: \"kubernetes.io/projected/d915c045-08fd-4ca6-a35d-0fa360cee77c-kube-api-access-b5zfc\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.687711 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.687689    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d915c045-08fd-4ca6-a35d-0fa360cee77c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.687944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.687714    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d915c045-08fd-4ca6-a35d-0fa360cee77c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.688406 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.688345    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d915c045-08fd-4ca6-a35d-0fa360cee77c-serving-certs-ca-bundle\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.688692 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.688617    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d915c045-08fd-4ca6-a35d-0fa360cee77c-metrics-client-ca\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.688801 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.688761    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d915c045-08fd-4ca6-a35d-0fa360cee77c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.690856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.690833    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d915c045-08fd-4ca6-a35d-0fa360cee77c-telemeter-client-tls\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.690989 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.690965    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d915c045-08fd-4ca6-a35d-0fa360cee77c-secret-telemeter-client\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.691035 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.691002    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d915c045-08fd-4ca6-a35d-0fa360cee77c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.691067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.691038    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d915c045-08fd-4ca6-a35d-0fa360cee77c-federate-client-tls\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.700731 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.700705    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5zfc\" (UniqueName: \"kubernetes.io/projected/d915c045-08fd-4ca6-a35d-0fa360cee77c-kube-api-access-b5zfc\") pod \"telemeter-client-7df55dc5f-g8p4f\" (UID: \"d915c045-08fd-4ca6-a35d-0fa360cee77c\") " pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:49.791860 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:49.791820    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"
Nov 02 09:35:50.079426 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:50.079396    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7df55dc5f-g8p4f"]
Nov 02 09:35:50.092321 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:50.092291    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd915c045_08fd_4ca6_a35d_0fa360cee77c.slice/crio-835cc14d543405d0013c130adbd1586f228f1cfca4654e4655272226eab2e014 WatchSource:0}: Error finding container 835cc14d543405d0013c130adbd1586f228f1cfca4654e4655272226eab2e014: Status 404 returned error can't find the container with id 835cc14d543405d0013c130adbd1586f228f1cfca4654e4655272226eab2e014
Nov 02 09:35:50.340620 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:50.340585    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f" event={"ID":"d915c045-08fd-4ca6-a35d-0fa360cee77c","Type":"ContainerStarted","Data":"835cc14d543405d0013c130adbd1586f228f1cfca4654e4655272226eab2e014"}
Nov 02 09:35:50.343574 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:50.343550    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh" event={"ID":"5f39265c-2507-4c75-969c-33cdaea8f677","Type":"ContainerStarted","Data":"3852f3280ac1347cde16155b8e69205b68632d7ca4ecea968801e16fbb48f862"}
Nov 02 09:35:50.343668 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:50.343602    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh" event={"ID":"5f39265c-2507-4c75-969c-33cdaea8f677","Type":"ContainerStarted","Data":"640d546ed93396b15c3a91c87a215c48235d24e72ec0deb0c527e99e0f0094e4"}
Nov 02 09:35:50.343668 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:50.343628    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh" event={"ID":"5f39265c-2507-4c75-969c-33cdaea8f677","Type":"ContainerStarted","Data":"31551a7451620106e2b9b19d63004185d808fc0d700b91599eaab7f85aae0aeb"}
Nov 02 09:35:50.343752 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:50.343726    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:50.370073 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:50.369986    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh" podStartSLOduration=1.5705704809999999 podStartE2EDuration="4.369973649s" totalImagesPullingTime="2.799403168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:50.369369381 +0000 UTC m=+27.307656180" watchObservedRunningTime="2026-11-02 09:35:50.369973649 +0000 UTC m=+27.308260422"
Nov 02 09:35:52.356308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:52.356278    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f" event={"ID":"d915c045-08fd-4ca6-a35d-0fa360cee77c","Type":"ContainerStarted","Data":"719698116b19de701ac10e85a972618d83086ebcf0d2438b2ea00ab68e0bdc75"}
Nov 02 09:35:53.310824 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:53.310798    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sjj6v"
Nov 02 09:35:54.365860 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:54.365824    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f" event={"ID":"d915c045-08fd-4ca6-a35d-0fa360cee77c","Type":"ContainerStarted","Data":"4301878a6e8078db415658d5f1e742ff2a946279d3a6066ad3ac448d47409739"}
Nov 02 09:35:54.366267 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:54.365890    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f" event={"ID":"d915c045-08fd-4ca6-a35d-0fa360cee77c","Type":"ContainerStarted","Data":"284d8e9efa336edf1bf21f38b0c7ee02bc941729d835294541817a852f1a318f"}
Nov 02 09:35:54.390477 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:54.390428    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7df55dc5f-g8p4f" podStartSLOduration=2.028479131 podStartE2EDuration="5.390413844s" totalImagesPullingTime="3.361934713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:54.389427308 +0000 UTC m=+31.327714098" watchObservedRunningTime="2026-11-02 09:35:54.390413844 +0000 UTC m=+31.328700618"
Nov 02 09:35:56.352757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:56.352730    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6d9cc5b7-z88nh"
Nov 02 09:35:56.486073 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:56.486038    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2gvpb"
Nov 02 09:35:58.136330 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.136293    4449 patch_prober.go:29] interesting pod/image-registry-84b96dbbff-vb64t container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 09:35:58.136802 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.136363    4449 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84b96dbbff-vb64t" podUID="67fc47bc-862f-4311-918f-2af036ebd4b4" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 09:35:58.173534 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.173502    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-g6wb2"]
Nov 02 09:35:58.175720 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.175703    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-g6wb2"
Nov 02 09:35:58.179658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.179638    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 09:35:58.179658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.179638    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-k464z\""
Nov 02 09:35:58.180386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.180367    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 09:35:58.187604 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.187580    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-g6wb2"]
Nov 02 09:35:58.253814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.253780    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpmh\" (UniqueName: \"kubernetes.io/projected/b30515be-06ba-4ccf-95fe-b449af52203c-kube-api-access-vrpmh\") pod \"downloads-6dd789c8b7-g6wb2\" (UID: \"b30515be-06ba-4ccf-95fe-b449af52203c\") " pod="openshift-console/downloads-6dd789c8b7-g6wb2"
Nov 02 09:35:58.253984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.253843    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b30515be-06ba-4ccf-95fe-b449af52203c-tmp\") pod \"downloads-6dd789c8b7-g6wb2\" (UID: \"b30515be-06ba-4ccf-95fe-b449af52203c\") " pod="openshift-console/downloads-6dd789c8b7-g6wb2"
Nov 02 09:35:58.354935 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.354904    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpmh\" (UniqueName: \"kubernetes.io/projected/b30515be-06ba-4ccf-95fe-b449af52203c-kube-api-access-vrpmh\") pod \"downloads-6dd789c8b7-g6wb2\" (UID: \"b30515be-06ba-4ccf-95fe-b449af52203c\") " pod="openshift-console/downloads-6dd789c8b7-g6wb2"
Nov 02 09:35:58.355129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.354962    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b30515be-06ba-4ccf-95fe-b449af52203c-tmp\") pod \"downloads-6dd789c8b7-g6wb2\" (UID: \"b30515be-06ba-4ccf-95fe-b449af52203c\") " pod="openshift-console/downloads-6dd789c8b7-g6wb2"
Nov 02 09:35:58.355366 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.355350    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b30515be-06ba-4ccf-95fe-b449af52203c-tmp\") pod \"downloads-6dd789c8b7-g6wb2\" (UID: \"b30515be-06ba-4ccf-95fe-b449af52203c\") " pod="openshift-console/downloads-6dd789c8b7-g6wb2"
Nov 02 09:35:58.367942 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.367919    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpmh\" (UniqueName: \"kubernetes.io/projected/b30515be-06ba-4ccf-95fe-b449af52203c-kube-api-access-vrpmh\") pod \"downloads-6dd789c8b7-g6wb2\" (UID: \"b30515be-06ba-4ccf-95fe-b449af52203c\") " pod="openshift-console/downloads-6dd789c8b7-g6wb2"
Nov 02 09:35:58.485942 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.485877    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-g6wb2"
Nov 02 09:35:58.759121 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:58.759033    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-g6wb2"]
Nov 02 09:35:58.763342 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:35:58.763309    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30515be_06ba_4ccf_95fe_b449af52203c.slice/crio-9c972a506325cc386f0ef2b20b4d895e08c9c00ee8cd4889cef865ebc3ab57a6 WatchSource:0}: Error finding container 9c972a506325cc386f0ef2b20b4d895e08c9c00ee8cd4889cef865ebc3ab57a6: Status 404 returned error can't find the container with id 9c972a506325cc386f0ef2b20b4d895e08c9c00ee8cd4889cef865ebc3ab57a6
Nov 02 09:35:59.383046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:35:59.383014    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-g6wb2" event={"ID":"b30515be-06ba-4ccf-95fe-b449af52203c","Type":"ContainerStarted","Data":"9c972a506325cc386f0ef2b20b4d895e08c9c00ee8cd4889cef865ebc3ab57a6"}
Nov 02 09:36:00.296966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:00.296940    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84b96dbbff-vb64t"
Nov 02 09:36:06.550945 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:36:06.550852    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:36:07.350830 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.350798    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58684db8b4-s7s99"]
Nov 02 09:36:07.354535 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.354513    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.363950 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.363927    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 09:36:07.363950 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.363930    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 09:36:07.364152 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.364138    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 09:36:07.366537 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.366478    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58684db8b4-s7s99"]
Nov 02 09:36:07.434567 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.434538    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/809c9d21-8fdd-40e9-80e6-21153c316b94-tmp\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.434567 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.434574    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-service-ca\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.434762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.434594    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-oauth-config\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.434762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.434724    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-console-config\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.434762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.434750    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-oauth-serving-cert\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.434917 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.434846    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-serving-cert\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.434917 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.434884    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4dqt\" (UniqueName: \"kubernetes.io/projected/809c9d21-8fdd-40e9-80e6-21153c316b94-kube-api-access-n4dqt\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.450753 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.450719    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 09:36:07.450931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.450869    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6wv98\""
Nov 02 09:36:07.450931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.450869    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 09:36:07.536054 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.536020    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-oauth-config\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.536258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.536070    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-console-config\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.536258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.536091    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-oauth-serving-cert\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.536258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.536161    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-serving-cert\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.536258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.536187    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4dqt\" (UniqueName: \"kubernetes.io/projected/809c9d21-8fdd-40e9-80e6-21153c316b94-kube-api-access-n4dqt\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.536258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.536208    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/809c9d21-8fdd-40e9-80e6-21153c316b94-tmp\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.536258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.536251    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-service-ca\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.537199 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.537162    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-oauth-serving-cert\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.537199 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.537178    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-console-config\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.537397 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.537166    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/809c9d21-8fdd-40e9-80e6-21153c316b94-tmp\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.537397 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.537243    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-service-ca\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.539087 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.539064    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-oauth-config\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.539606 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.539579    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-serving-cert\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.548998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.548973    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4dqt\" (UniqueName: \"kubernetes.io/projected/809c9d21-8fdd-40e9-80e6-21153c316b94-kube-api-access-n4dqt\") pod \"console-58684db8b4-s7s99\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") " pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.668944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.668856    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:07.951700 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:07.951626    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58684db8b4-s7s99"]
Nov 02 09:36:07.955100 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:36:07.955071    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809c9d21_8fdd_40e9_80e6_21153c316b94.slice/crio-f99143255e6c1476038771777015923561ddc6277aa95ac37f19515e6d32e5aa WatchSource:0}: Error finding container f99143255e6c1476038771777015923561ddc6277aa95ac37f19515e6d32e5aa: Status 404 returned error can't find the container with id f99143255e6c1476038771777015923561ddc6277aa95ac37f19515e6d32e5aa
Nov 02 09:36:08.416933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:08.416900    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58684db8b4-s7s99" event={"ID":"809c9d21-8fdd-40e9-80e6-21153c316b94","Type":"ContainerStarted","Data":"f99143255e6c1476038771777015923561ddc6277aa95ac37f19515e6d32e5aa"}
Nov 02 09:36:11.431804 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:11.431714    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58684db8b4-s7s99" event={"ID":"809c9d21-8fdd-40e9-80e6-21153c316b94","Type":"ContainerStarted","Data":"d66e63c4d87731d3ab843378c9b4b21fddc5b6ccf9c288c4a091df654274f9d0"}
Nov 02 09:36:11.454369 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:11.454311    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58684db8b4-s7s99" podStartSLOduration=1.3483617909999999 podStartE2EDuration="4.454296937s" totalImagesPullingTime="3.105935146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:36:11.45317428 +0000 UTC m=+48.391461085" watchObservedRunningTime="2026-11-02 09:36:11.454296937 +0000 UTC m=+48.392583712"
Nov 02 09:36:15.450641 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:15.450603    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-g6wb2" event={"ID":"b30515be-06ba-4ccf-95fe-b449af52203c","Type":"ContainerStarted","Data":"bd913092b56ea62d9a6f4fe963946c759e465dd9ba0b15a17b8c499a26b56819"}
Nov 02 09:36:15.451120 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:15.450836    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-g6wb2"
Nov 02 09:36:15.465458 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:15.465426    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-g6wb2"
Nov 02 09:36:15.473335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:15.473285    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-g6wb2" podStartSLOduration=1.693278681 podStartE2EDuration="17.473268707s" totalImagesPullingTime="15.779990026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:36:15.471966044 +0000 UTC m=+52.410252844" watchObservedRunningTime="2026-11-02 09:36:15.473268707 +0000 UTC m=+52.411555485"
Nov 02 09:36:17.669302 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:17.669262    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:17.669769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:17.669318    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:17.675137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:17.675114    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:18.390725 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.390692    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b9d6bc84d-jsp8x"]
Nov 02 09:36:18.421999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.421968    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b9d6bc84d-jsp8x"]
Nov 02 09:36:18.422140 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.422094    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.430768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.430738    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 02 09:36:18.477251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.477205    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:18.535661 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.535622    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-serving-cert\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.535837 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.535693    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-trusted-ca-bundle\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.535977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.535937    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-service-ca\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.536094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.536051    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-oauth-config\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.536094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.536081    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5001d46-97a2-4947-8950-00a056377a3e-tmp\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.536237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.536127    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-console-config\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.536237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.536152    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtp5w\" (UniqueName: \"kubernetes.io/projected/f5001d46-97a2-4947-8950-00a056377a3e-kube-api-access-wtp5w\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.536347 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.536272    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-oauth-serving-cert\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.637462 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.637420    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-serving-cert\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.637648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.637469    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-trusted-ca-bundle\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.637648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.637505    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-service-ca\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.637648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.637531    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-oauth-config\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.637648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.637555    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5001d46-97a2-4947-8950-00a056377a3e-tmp\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.637648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.637603    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-console-config\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.637648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.637631    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtp5w\" (UniqueName: \"kubernetes.io/projected/f5001d46-97a2-4947-8950-00a056377a3e-kube-api-access-wtp5w\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.638012 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.637688    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-oauth-serving-cert\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.638378 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.638347    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-service-ca\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.638378 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.638367    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-oauth-serving-cert\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.638663 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.638634    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5001d46-97a2-4947-8950-00a056377a3e-tmp\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.638774 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.638692    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-console-config\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.639497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.639474    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-trusted-ca-bundle\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.640501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.640471    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-oauth-config\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.641557 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.641513    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-serving-cert\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.651169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.651147    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtp5w\" (UniqueName: \"kubernetes.io/projected/f5001d46-97a2-4947-8950-00a056377a3e-kube-api-access-wtp5w\") pod \"console-5b9d6bc84d-jsp8x\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") " pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:18.748430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:18.748396    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:19.048524 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:19.048422    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b9d6bc84d-jsp8x"]
Nov 02 09:36:19.051732 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:36:19.051707    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5001d46_97a2_4947_8950_00a056377a3e.slice/crio-4ab2a1bd0a8863761e2b4ec09bbc99a2fd266656a97267c278404e1652e21bc9 WatchSource:0}: Error finding container 4ab2a1bd0a8863761e2b4ec09bbc99a2fd266656a97267c278404e1652e21bc9: Status 404 returned error can't find the container with id 4ab2a1bd0a8863761e2b4ec09bbc99a2fd266656a97267c278404e1652e21bc9
Nov 02 09:36:19.470240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:19.470121    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9d6bc84d-jsp8x" event={"ID":"f5001d46-97a2-4947-8950-00a056377a3e","Type":"ContainerStarted","Data":"ec2e751e31c010af5c14fb209304b609f30f0a1b4e1158febfb60237b1641933"}
Nov 02 09:36:19.470240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:19.470184    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9d6bc84d-jsp8x" event={"ID":"f5001d46-97a2-4947-8950-00a056377a3e","Type":"ContainerStarted","Data":"4ab2a1bd0a8863761e2b4ec09bbc99a2fd266656a97267c278404e1652e21bc9"}
Nov 02 09:36:28.749368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:28.749326    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:28.749860 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:28.749386    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:28.754206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:28.754176    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:28.775004 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:28.774949    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b9d6bc84d-jsp8x" podStartSLOduration=10.774934524 podStartE2EDuration="10.774934524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:36:19.492314573 +0000 UTC m=+56.430601390" watchObservedRunningTime="2026-11-02 09:36:28.774934524 +0000 UTC m=+65.713221298"
Nov 02 09:36:29.510683 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:29.510659    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:36:29.562187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:29.562151    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58684db8b4-s7s99"]
Nov 02 09:36:36.546954 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:36:36.546917    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:36:46.830195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.830154    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 09:36:46.864289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.864246    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 09:36:46.864438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.864347    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:46.869912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.869793    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 09:36:46.869912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.869810    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 09:36:46.869912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.869815    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 09:36:46.869912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.869855    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 09:36:46.869912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.869855    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 09:36:46.869912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.869875    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 09:36:46.869912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.869875    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 09:36:46.870392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.869924    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-99zt2\""
Nov 02 09:36:46.870815 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.870787    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 09:36:46.871421 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.871109    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 09:36:46.986540 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.986506    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:46.986706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.986563    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:46.986706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.986630    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-config-volume\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:46.986706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.986679    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:46.986706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.986702    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d60d3785-1f6c-405a-8785-048827802da2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:46.986834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.986720    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d60d3785-1f6c-405a-8785-048827802da2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:46.986834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.986790    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d60d3785-1f6c-405a-8785-048827802da2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:46.986895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.986829    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9k6g\" (UniqueName: \"kubernetes.io/projected/d60d3785-1f6c-405a-8785-048827802da2-kube-api-access-j9k6g\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:46.986895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.986879    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-web-config\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:46.986954 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.986910    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d60d3785-1f6c-405a-8785-048827802da2-config-out\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:46.986954 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.986941    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:46.987014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.986981    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d60d3785-1f6c-405a-8785-048827802da2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:46.987014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:46.987003    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.088441 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088369    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.088441 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088411    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.088441 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088429    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-config-volume\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.088722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088448    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.088722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088468    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d60d3785-1f6c-405a-8785-048827802da2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.088722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088491    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d60d3785-1f6c-405a-8785-048827802da2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.088722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088520    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d60d3785-1f6c-405a-8785-048827802da2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.088722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088548    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9k6g\" (UniqueName: \"kubernetes.io/projected/d60d3785-1f6c-405a-8785-048827802da2-kube-api-access-j9k6g\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.088722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088592    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-web-config\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.088722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088627    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d60d3785-1f6c-405a-8785-048827802da2-config-out\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.088722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088645    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.088722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088673    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d60d3785-1f6c-405a-8785-048827802da2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.088722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088701    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.089248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.088963    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d60d3785-1f6c-405a-8785-048827802da2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.089994 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.089969    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d60d3785-1f6c-405a-8785-048827802da2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.091756 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.091731    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d60d3785-1f6c-405a-8785-048827802da2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.092110 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.092073    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.092282 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.092255    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.092372 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.092308    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.092596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.092556    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-config-volume\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.092800 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.092779    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d60d3785-1f6c-405a-8785-048827802da2-config-out\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.092868 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.092797    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.093133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.093113    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.093174 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.093164    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d60d3785-1f6c-405a-8785-048827802da2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.093581 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.093565    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d60d3785-1f6c-405a-8785-048827802da2-web-config\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.103869 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.103846    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9k6g\" (UniqueName: \"kubernetes.io/projected/d60d3785-1f6c-405a-8785-048827802da2-kube-api-access-j9k6g\") pod \"alertmanager-main-0\" (UID: \"d60d3785-1f6c-405a-8785-048827802da2\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.270357 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.270319    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 09:36:47.584975 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:47.584949    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 09:36:47.587858 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:36:47.587830    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60d3785_1f6c_405a_8785_048827802da2.slice/crio-38140ea12c3dedb4d70ff0932e22052492cb835fed0169fde1b176ab7c14471c WatchSource:0}: Error finding container 38140ea12c3dedb4d70ff0932e22052492cb835fed0169fde1b176ab7c14471c: Status 404 returned error can't find the container with id 38140ea12c3dedb4d70ff0932e22052492cb835fed0169fde1b176ab7c14471c
Nov 02 09:36:48.573504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:48.573469    4449 generic.go:356] "Generic (PLEG): container finished" podID="d60d3785-1f6c-405a-8785-048827802da2" containerID="c3f2c0ff39766f3b0cee50fb776a960f3d90ae5279457e56782bafc59f04e08c" exitCode=0
Nov 02 09:36:48.573896 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:48.573530    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d60d3785-1f6c-405a-8785-048827802da2","Type":"ContainerDied","Data":"c3f2c0ff39766f3b0cee50fb776a960f3d90ae5279457e56782bafc59f04e08c"}
Nov 02 09:36:48.573896 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:48.573580    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d60d3785-1f6c-405a-8785-048827802da2","Type":"ContainerStarted","Data":"38140ea12c3dedb4d70ff0932e22052492cb835fed0169fde1b176ab7c14471c"}
Nov 02 09:36:51.591452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:51.591417    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d60d3785-1f6c-405a-8785-048827802da2","Type":"ContainerStarted","Data":"dcbcfce6a35e09a2ae31e4762b1104654ffc6a35d1f6c830d7dd67fb2c370ae3"}
Nov 02 09:36:51.591833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:51.591523    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d60d3785-1f6c-405a-8785-048827802da2","Type":"ContainerStarted","Data":"6c7aedfe048a2a8862b66a88831451b472ae6d7c67e3c35dea7feef8c0e34653"}
Nov 02 09:36:51.591833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:51.591567    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d60d3785-1f6c-405a-8785-048827802da2","Type":"ContainerStarted","Data":"84f3c2c5fd8f1446b8801882c7a68eba1e5dcdaff6f58426c14fb8853f0b895b"}
Nov 02 09:36:51.591833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:51.591604    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d60d3785-1f6c-405a-8785-048827802da2","Type":"ContainerStarted","Data":"8979c3ce7e3c702c2628c0875223661a6f428968e032023fdb1bb0bd85512b78"}
Nov 02 09:36:52.165323 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.165284    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-pgbtq-2c6rz"]
Nov 02 09:36:52.165517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.165495    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-pgbtq-2c6rz" podUID="86739438-2924-4642-9d34-ed545e1739ba" containerName="insights-gathering" containerID="cri-o://5d6fb61fed161d8ddc64effbc014568a83458f25561283646424eeeff4d8c0f6" gracePeriod=30
Nov 02 09:36:52.433981 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.433896    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:36:52.533096 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.533058    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86739438-2924-4642-9d34-ed545e1739ba-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/86739438-2924-4642-9d34-ed545e1739ba-archives-path\") pod \"86739438-2924-4642-9d34-ed545e1739ba\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") "
Nov 02 09:36:52.533297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.533116    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-service-ca-bundle\") pod \"86739438-2924-4642-9d34-ed545e1739ba\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") "
Nov 02 09:36:52.533297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.533165    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-trusted-ca-bundle\") pod \"86739438-2924-4642-9d34-ed545e1739ba\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") "
Nov 02 09:36:52.533297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.533201    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86739438-2924-4642-9d34-ed545e1739ba-kube-api-access-lsmrc\" (UniqueName: \"kubernetes.io/projected/86739438-2924-4642-9d34-ed545e1739ba-kube-api-access-lsmrc\") pod \"86739438-2924-4642-9d34-ed545e1739ba\" (UID: \"86739438-2924-4642-9d34-ed545e1739ba\") "
Nov 02 09:36:52.533501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.533474    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86739438-2924-4642-9d34-ed545e1739ba-archives-path" pod "86739438-2924-4642-9d34-ed545e1739ba" (UID: "86739438-2924-4642-9d34-ed545e1739ba"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:36:52.533618 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.533591    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-service-ca-bundle" pod "86739438-2924-4642-9d34-ed545e1739ba" (UID: "86739438-2924-4642-9d34-ed545e1739ba"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:36:52.533670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.533600    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-trusted-ca-bundle" pod "86739438-2924-4642-9d34-ed545e1739ba" (UID: "86739438-2924-4642-9d34-ed545e1739ba"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:36:52.535760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.535735    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86739438-2924-4642-9d34-ed545e1739ba-kube-api-access-lsmrc" pod "86739438-2924-4642-9d34-ed545e1739ba" (UID: "86739438-2924-4642-9d34-ed545e1739ba"). InnerVolumeSpecName "kube-api-access-lsmrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:36:52.598773 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.598733    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d60d3785-1f6c-405a-8785-048827802da2","Type":"ContainerStarted","Data":"23f7eb9bf7eb86365ee69edd5b70265fb8305a793fbafd5705786c9fb167fd49"}
Nov 02 09:36:52.599289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.598807    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d60d3785-1f6c-405a-8785-048827802da2","Type":"ContainerStarted","Data":"94129aee14aaea6a973f65ffc43424e33639ea49c782041fd59996b4d1cd8c6b"}
Nov 02 09:36:52.600042 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.600018    4449 generic.go:356] "Generic (PLEG): container finished" podID="86739438-2924-4642-9d34-ed545e1739ba" containerID="5d6fb61fed161d8ddc64effbc014568a83458f25561283646424eeeff4d8c0f6" exitCode=2
Nov 02 09:36:52.600113 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.600066    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pgbtq-2c6rz"
Nov 02 09:36:52.600148 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.600103    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pgbtq-2c6rz" event={"ID":"86739438-2924-4642-9d34-ed545e1739ba","Type":"ContainerDied","Data":"5d6fb61fed161d8ddc64effbc014568a83458f25561283646424eeeff4d8c0f6"}
Nov 02 09:36:52.600180 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.600154    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pgbtq-2c6rz" event={"ID":"86739438-2924-4642-9d34-ed545e1739ba","Type":"ContainerDied","Data":"1cc09061c97b92a0cbebcd9d3aedb18b9004d60918452703e9c14d89714ae540"}
Nov 02 09:36:52.600238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.600182    4449 scope.go:122] "RemoveContainer" containerID="5d6fb61fed161d8ddc64effbc014568a83458f25561283646424eeeff4d8c0f6"
Nov 02 09:36:52.609829 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.609718    4449 scope.go:122] "RemoveContainer" containerID="5d6fb61fed161d8ddc64effbc014568a83458f25561283646424eeeff4d8c0f6"
Nov 02 09:36:52.610585 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:36:52.610482    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6fb61fed161d8ddc64effbc014568a83458f25561283646424eeeff4d8c0f6\": container with ID starting with 5d6fb61fed161d8ddc64effbc014568a83458f25561283646424eeeff4d8c0f6 not found: ID does not exist" containerID="5d6fb61fed161d8ddc64effbc014568a83458f25561283646424eeeff4d8c0f6"
Nov 02 09:36:52.610585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.610523    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6fb61fed161d8ddc64effbc014568a83458f25561283646424eeeff4d8c0f6"} err="failed to get container status \"5d6fb61fed161d8ddc64effbc014568a83458f25561283646424eeeff4d8c0f6\": rpc error: code = NotFound desc = could not find container \"5d6fb61fed161d8ddc64effbc014568a83458f25561283646424eeeff4d8c0f6\": container with ID starting with 5d6fb61fed161d8ddc64effbc014568a83458f25561283646424eeeff4d8c0f6 not found: ID does not exist"
Nov 02 09:36:52.631448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.631387    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.034317018 podStartE2EDuration="6.631370858s" totalImagesPullingTime="2.58259235s" totalInitContainerRuntime="14.46149ms" isStatefulPod=false podCreationTimestamp="2026-11-02 09:36:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:36:52.630195272 +0000 UTC m=+89.568482066" watchObservedRunningTime="2026-11-02 09:36:52.631370858 +0000 UTC m=+89.569657655"
Nov 02 09:36:52.633716 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.633676    4449 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-trusted-ca-bundle\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:52.633716 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.633715    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsmrc\" (UniqueName: \"kubernetes.io/projected/86739438-2924-4642-9d34-ed545e1739ba-kube-api-access-lsmrc\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:52.633886 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.633729    4449 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/86739438-2924-4642-9d34-ed545e1739ba-archives-path\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:52.633886 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.633741    4449 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86739438-2924-4642-9d34-ed545e1739ba-service-ca-bundle\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:52.643738 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.643705    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-pgbtq-2c6rz"]
Nov 02 09:36:52.647712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:52.647683    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-pgbtq-2c6rz"]
Nov 02 09:36:53.204352 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:53.204317    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86739438-2924-4642-9d34-ed545e1739ba" path="/var/lib/kubelet/pods/86739438-2924-4642-9d34-ed545e1739ba/volumes"
Nov 02 09:36:53.916656 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:53.916614    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58bc4dbc95-6p2qz"]
Nov 02 09:36:53.917134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:53.917049    4449 cpu_manager.go:414] "containerMap: removing container" podUID="86739438-2924-4642-9d34-ed545e1739ba" containerName="insights-gathering"
Nov 02 09:36:53.917134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:53.917067    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86739438-2924-4642-9d34-ed545e1739ba" containerName="insights-gathering"
Nov 02 09:36:53.917286 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:53.917167    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="86739438-2924-4642-9d34-ed545e1739ba" containerName="insights-gathering"
Nov 02 09:36:53.944090 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:53.944045    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58bc4dbc95-6p2qz"]
Nov 02 09:36:53.944270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:53.944146    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.044848 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.044813    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-tmp\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.044848 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.044855    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zv7\" (UniqueName: \"kubernetes.io/projected/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-kube-api-access-p9zv7\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.045126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.044873    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-service-ca\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.045126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.044959    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-trusted-ca-bundle\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.045126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.044993    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-console-config\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.045126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.045043    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-console-oauth-config\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.045126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.045079    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-oauth-serving-cert\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.045126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.045120    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-console-serving-cert\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.146528 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.146485    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-oauth-serving-cert\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.146528 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.146531    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-console-serving-cert\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.146771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.146583    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-tmp\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.146771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.146618    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zv7\" (UniqueName: \"kubernetes.io/projected/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-kube-api-access-p9zv7\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.146771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.146640    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-service-ca\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.146771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.146675    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-trusted-ca-bundle\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.146771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.146697    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-console-config\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.146771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.146726    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-console-oauth-config\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.147274 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.147247    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-tmp\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.147452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.147428    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-service-ca\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.147517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.147463    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-oauth-serving-cert\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.147517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.147469    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-console-config\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.147712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.147695    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-trusted-ca-bundle\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.150037 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.150010    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-console-oauth-config\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.150135 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.150096    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-console-serving-cert\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.160036 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.160006    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zv7\" (UniqueName: \"kubernetes.io/projected/accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37-kube-api-access-p9zv7\") pod \"console-58bc4dbc95-6p2qz\" (UID: \"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37\") " pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.268678 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.268569    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:36:54.573033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.572998    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58bc4dbc95-6p2qz"]
Nov 02 09:36:54.576133 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:36:54.576101    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccacbf0_bbb4_4ffb_9ebc_dcd26fc62a37.slice/crio-2ee76f531d9c2b9039f4409878bc2169261a8ee8962936634264d98f8a460023 WatchSource:0}: Error finding container 2ee76f531d9c2b9039f4409878bc2169261a8ee8962936634264d98f8a460023: Status 404 returned error can't find the container with id 2ee76f531d9c2b9039f4409878bc2169261a8ee8962936634264d98f8a460023
Nov 02 09:36:54.582712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.582674    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-58684db8b4-s7s99" podUID="809c9d21-8fdd-40e9-80e6-21153c316b94" containerName="console" containerID="cri-o://d66e63c4d87731d3ab843378c9b4b21fddc5b6ccf9c288c4a091df654274f9d0" gracePeriod=15
Nov 02 09:36:54.611049 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.611018    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58bc4dbc95-6p2qz" event={"ID":"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37","Type":"ContainerStarted","Data":"2ee76f531d9c2b9039f4409878bc2169261a8ee8962936634264d98f8a460023"}
Nov 02 09:36:54.847899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.847863    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:54.954902 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.954866    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809c9d21-8fdd-40e9-80e6-21153c316b94-kube-api-access-n4dqt\" (UniqueName: \"kubernetes.io/projected/809c9d21-8fdd-40e9-80e6-21153c316b94-kube-api-access-n4dqt\") pod \"809c9d21-8fdd-40e9-80e6-21153c316b94\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") "
Nov 02 09:36:54.955440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.954913    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-serving-cert\") pod \"809c9d21-8fdd-40e9-80e6-21153c316b94\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") "
Nov 02 09:36:54.955440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.954943    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-oauth-serving-cert\") pod \"809c9d21-8fdd-40e9-80e6-21153c316b94\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") "
Nov 02 09:36:54.955440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.954977    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809c9d21-8fdd-40e9-80e6-21153c316b94-tmp\" (UniqueName: \"kubernetes.io/empty-dir/809c9d21-8fdd-40e9-80e6-21153c316b94-tmp\") pod \"809c9d21-8fdd-40e9-80e6-21153c316b94\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") "
Nov 02 09:36:54.955440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.955095    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-oauth-config\") pod \"809c9d21-8fdd-40e9-80e6-21153c316b94\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") "
Nov 02 09:36:54.955440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.955175    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-console-config\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-console-config\") pod \"809c9d21-8fdd-40e9-80e6-21153c316b94\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") "
Nov 02 09:36:54.955440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.955243    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-service-ca\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-service-ca\") pod \"809c9d21-8fdd-40e9-80e6-21153c316b94\" (UID: \"809c9d21-8fdd-40e9-80e6-21153c316b94\") "
Nov 02 09:36:54.955440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.955265    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809c9d21-8fdd-40e9-80e6-21153c316b94-tmp" pod "809c9d21-8fdd-40e9-80e6-21153c316b94" (UID: "809c9d21-8fdd-40e9-80e6-21153c316b94"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:36:54.955866 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.955539    4449 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/809c9d21-8fdd-40e9-80e6-21153c316b94-tmp\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:54.955866 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.955563    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-console-config" pod "809c9d21-8fdd-40e9-80e6-21153c316b94" (UID: "809c9d21-8fdd-40e9-80e6-21153c316b94"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:36:54.955866 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.955626    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-service-ca" pod "809c9d21-8fdd-40e9-80e6-21153c316b94" (UID: "809c9d21-8fdd-40e9-80e6-21153c316b94"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:36:54.955866 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.955686    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-oauth-serving-cert" pod "809c9d21-8fdd-40e9-80e6-21153c316b94" (UID: "809c9d21-8fdd-40e9-80e6-21153c316b94"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:36:54.957542 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.957515    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-serving-cert" pod "809c9d21-8fdd-40e9-80e6-21153c316b94" (UID: "809c9d21-8fdd-40e9-80e6-21153c316b94"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:36:54.957653 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.957546    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-oauth-config" pod "809c9d21-8fdd-40e9-80e6-21153c316b94" (UID: "809c9d21-8fdd-40e9-80e6-21153c316b94"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:36:54.957653 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:54.957566    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809c9d21-8fdd-40e9-80e6-21153c316b94-kube-api-access-n4dqt" pod "809c9d21-8fdd-40e9-80e6-21153c316b94" (UID: "809c9d21-8fdd-40e9-80e6-21153c316b94"). InnerVolumeSpecName "kube-api-access-n4dqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:36:55.056910 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.056870    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4dqt\" (UniqueName: \"kubernetes.io/projected/809c9d21-8fdd-40e9-80e6-21153c316b94-kube-api-access-n4dqt\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:55.056910 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.056900    4449 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-serving-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:55.056910 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.056910    4449 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-oauth-serving-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:55.056910 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.056920    4449 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/809c9d21-8fdd-40e9-80e6-21153c316b94-console-oauth-config\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:55.057162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.056929    4449 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-console-config\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:55.057162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.056937    4449 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/809c9d21-8fdd-40e9-80e6-21153c316b94-service-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:55.616469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.616432    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58bc4dbc95-6p2qz" event={"ID":"accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37","Type":"ContainerStarted","Data":"ee996133fb0cd677533deed26a771da62e2a5d5da144410e30fefd3a4c1a0e9d"}
Nov 02 09:36:55.617800 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.617775    4449 generic.go:356] "Generic (PLEG): container finished" podID="809c9d21-8fdd-40e9-80e6-21153c316b94" containerID="d66e63c4d87731d3ab843378c9b4b21fddc5b6ccf9c288c4a091df654274f9d0" exitCode=0
Nov 02 09:36:55.617927 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.617818    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58684db8b4-s7s99"
Nov 02 09:36:55.617927 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.617823    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58684db8b4-s7s99" event={"ID":"809c9d21-8fdd-40e9-80e6-21153c316b94","Type":"ContainerDied","Data":"d66e63c4d87731d3ab843378c9b4b21fddc5b6ccf9c288c4a091df654274f9d0"}
Nov 02 09:36:55.617927 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.617855    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58684db8b4-s7s99" event={"ID":"809c9d21-8fdd-40e9-80e6-21153c316b94","Type":"ContainerDied","Data":"f99143255e6c1476038771777015923561ddc6277aa95ac37f19515e6d32e5aa"}
Nov 02 09:36:55.617927 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.617873    4449 scope.go:122] "RemoveContainer" containerID="d66e63c4d87731d3ab843378c9b4b21fddc5b6ccf9c288c4a091df654274f9d0"
Nov 02 09:36:55.626520 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.626497    4449 scope.go:122] "RemoveContainer" containerID="d66e63c4d87731d3ab843378c9b4b21fddc5b6ccf9c288c4a091df654274f9d0"
Nov 02 09:36:55.626807 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:36:55.626788    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66e63c4d87731d3ab843378c9b4b21fddc5b6ccf9c288c4a091df654274f9d0\": container with ID starting with d66e63c4d87731d3ab843378c9b4b21fddc5b6ccf9c288c4a091df654274f9d0 not found: ID does not exist" containerID="d66e63c4d87731d3ab843378c9b4b21fddc5b6ccf9c288c4a091df654274f9d0"
Nov 02 09:36:55.626848 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.626822    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66e63c4d87731d3ab843378c9b4b21fddc5b6ccf9c288c4a091df654274f9d0"} err="failed to get container status \"d66e63c4d87731d3ab843378c9b4b21fddc5b6ccf9c288c4a091df654274f9d0\": rpc error: code = NotFound desc = could not find container \"d66e63c4d87731d3ab843378c9b4b21fddc5b6ccf9c288c4a091df654274f9d0\": container with ID starting with d66e63c4d87731d3ab843378c9b4b21fddc5b6ccf9c288c4a091df654274f9d0 not found: ID does not exist"
Nov 02 09:36:55.638188 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.638140    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58bc4dbc95-6p2qz" podStartSLOduration=2.638125726 podStartE2EDuration="2.638125726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:36:55.637328472 +0000 UTC m=+92.575615275" watchObservedRunningTime="2026-11-02 09:36:55.638125726 +0000 UTC m=+92.576412579"
Nov 02 09:36:55.650171 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.650138    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58684db8b4-s7s99"]
Nov 02 09:36:55.654297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:55.654271    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58684db8b4-s7s99"]
Nov 02 09:36:57.204395 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:36:57.204364    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809c9d21-8fdd-40e9-80e6-21153c316b94" path="/var/lib/kubelet/pods/809c9d21-8fdd-40e9-80e6-21153c316b94/volumes"
Nov 02 09:37:04.268944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:04.268904    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:37:04.269404 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:04.268993    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:37:04.273777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:04.273754    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:37:04.657306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:04.657274    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58bc4dbc95-6p2qz"
Nov 02 09:37:04.709987 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:04.709948    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b9d6bc84d-jsp8x"]
Nov 02 09:37:06.545833 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:37:06.545794    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:37:29.730543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:29.730481    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5b9d6bc84d-jsp8x" podUID="f5001d46-97a2-4947-8950-00a056377a3e" containerName="console" containerID="cri-o://ec2e751e31c010af5c14fb209304b609f30f0a1b4e1158febfb60237b1641933" gracePeriod=15
Nov 02 09:37:29.965481 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:29.965454    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:37:30.036016 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.035987    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5001d46-97a2-4947-8950-00a056377a3e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5001d46-97a2-4947-8950-00a056377a3e-tmp\") pod \"f5001d46-97a2-4947-8950-00a056377a3e\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") "
Nov 02 09:37:30.036197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.036035    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-service-ca\") pod \"f5001d46-97a2-4947-8950-00a056377a3e\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") "
Nov 02 09:37:30.036197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.036053    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-oauth-config\") pod \"f5001d46-97a2-4947-8950-00a056377a3e\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") "
Nov 02 09:37:30.036197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.036092    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-trusted-ca-bundle\") pod \"f5001d46-97a2-4947-8950-00a056377a3e\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") "
Nov 02 09:37:30.036197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.036109    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5001d46-97a2-4947-8950-00a056377a3e-kube-api-access-wtp5w\" (UniqueName: \"kubernetes.io/projected/f5001d46-97a2-4947-8950-00a056377a3e-kube-api-access-wtp5w\") pod \"f5001d46-97a2-4947-8950-00a056377a3e\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") "
Nov 02 09:37:30.036197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.036144    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-console-config\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-console-config\") pod \"f5001d46-97a2-4947-8950-00a056377a3e\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") "
Nov 02 09:37:30.036197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.036182    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-serving-cert\") pod \"f5001d46-97a2-4947-8950-00a056377a3e\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") "
Nov 02 09:37:30.036776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.036208    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-oauth-serving-cert\") pod \"f5001d46-97a2-4947-8950-00a056377a3e\" (UID: \"f5001d46-97a2-4947-8950-00a056377a3e\") "
Nov 02 09:37:30.036776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.036370    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5001d46-97a2-4947-8950-00a056377a3e-tmp" pod "f5001d46-97a2-4947-8950-00a056377a3e" (UID: "f5001d46-97a2-4947-8950-00a056377a3e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:37:30.036776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.036510    4449 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5001d46-97a2-4947-8950-00a056377a3e-tmp\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:37:30.036776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.036548    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-service-ca" pod "f5001d46-97a2-4947-8950-00a056377a3e" (UID: "f5001d46-97a2-4947-8950-00a056377a3e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:37:30.036776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.036673    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-trusted-ca-bundle" pod "f5001d46-97a2-4947-8950-00a056377a3e" (UID: "f5001d46-97a2-4947-8950-00a056377a3e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:37:30.036776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.036734    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-console-config" pod "f5001d46-97a2-4947-8950-00a056377a3e" (UID: "f5001d46-97a2-4947-8950-00a056377a3e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:37:30.037040 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.036954    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-oauth-serving-cert" pod "f5001d46-97a2-4947-8950-00a056377a3e" (UID: "f5001d46-97a2-4947-8950-00a056377a3e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:37:30.038500 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.038481    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-oauth-config" pod "f5001d46-97a2-4947-8950-00a056377a3e" (UID: "f5001d46-97a2-4947-8950-00a056377a3e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:37:30.039109 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.039081    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5001d46-97a2-4947-8950-00a056377a3e-kube-api-access-wtp5w" pod "f5001d46-97a2-4947-8950-00a056377a3e" (UID: "f5001d46-97a2-4947-8950-00a056377a3e"). InnerVolumeSpecName "kube-api-access-wtp5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:37:30.039109 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.039090    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-serving-cert" pod "f5001d46-97a2-4947-8950-00a056377a3e" (UID: "f5001d46-97a2-4947-8950-00a056377a3e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:37:30.137155 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.137113    4449 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-service-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:37:30.137155 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.137153    4449 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-oauth-config\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:37:30.137155 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.137165    4449 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-trusted-ca-bundle\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:37:30.137404 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.137173    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtp5w\" (UniqueName: \"kubernetes.io/projected/f5001d46-97a2-4947-8950-00a056377a3e-kube-api-access-wtp5w\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:37:30.137404 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.137183    4449 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-console-config\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:37:30.137404 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.137192    4449 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5001d46-97a2-4947-8950-00a056377a3e-console-serving-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:37:30.137404 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.137200    4449 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5001d46-97a2-4947-8950-00a056377a3e-oauth-serving-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:37:30.747833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.747799    4449 generic.go:356] "Generic (PLEG): container finished" podID="f5001d46-97a2-4947-8950-00a056377a3e" containerID="ec2e751e31c010af5c14fb209304b609f30f0a1b4e1158febfb60237b1641933" exitCode=0
Nov 02 09:37:30.748334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.747849    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9d6bc84d-jsp8x"
Nov 02 09:37:30.748334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.747871    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9d6bc84d-jsp8x" event={"ID":"f5001d46-97a2-4947-8950-00a056377a3e","Type":"ContainerDied","Data":"ec2e751e31c010af5c14fb209304b609f30f0a1b4e1158febfb60237b1641933"}
Nov 02 09:37:30.748334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.747914    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9d6bc84d-jsp8x" event={"ID":"f5001d46-97a2-4947-8950-00a056377a3e","Type":"ContainerDied","Data":"4ab2a1bd0a8863761e2b4ec09bbc99a2fd266656a97267c278404e1652e21bc9"}
Nov 02 09:37:30.748334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.747933    4449 scope.go:122] "RemoveContainer" containerID="ec2e751e31c010af5c14fb209304b609f30f0a1b4e1158febfb60237b1641933"
Nov 02 09:37:30.756785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.756765    4449 scope.go:122] "RemoveContainer" containerID="ec2e751e31c010af5c14fb209304b609f30f0a1b4e1158febfb60237b1641933"
Nov 02 09:37:30.757089 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:37:30.757066    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2e751e31c010af5c14fb209304b609f30f0a1b4e1158febfb60237b1641933\": container with ID starting with ec2e751e31c010af5c14fb209304b609f30f0a1b4e1158febfb60237b1641933 not found: ID does not exist" containerID="ec2e751e31c010af5c14fb209304b609f30f0a1b4e1158febfb60237b1641933"
Nov 02 09:37:30.757157 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.757099    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2e751e31c010af5c14fb209304b609f30f0a1b4e1158febfb60237b1641933"} err="failed to get container status \"ec2e751e31c010af5c14fb209304b609f30f0a1b4e1158febfb60237b1641933\": rpc error: code = NotFound desc = could not find container \"ec2e751e31c010af5c14fb209304b609f30f0a1b4e1158febfb60237b1641933\": container with ID starting with ec2e751e31c010af5c14fb209304b609f30f0a1b4e1158febfb60237b1641933 not found: ID does not exist"
Nov 02 09:37:30.769111 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.769082    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b9d6bc84d-jsp8x"]
Nov 02 09:37:30.773234 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:30.773196    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b9d6bc84d-jsp8x"]
Nov 02 09:37:31.203201 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:37:31.203165    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5001d46-97a2-4947-8950-00a056377a3e" path="/var/lib/kubelet/pods/f5001d46-97a2-4947-8950-00a056377a3e/volumes"
Nov 02 09:37:36.545126 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:37:36.545044    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:38:06.545166 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:38:06.545122    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:38:36.545160 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:38:36.545120    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:38:47.252880 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.252845    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"]
Nov 02 09:38:47.253383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.253148    4449 cpu_manager.go:414] "containerMap: removing container" podUID="809c9d21-8fdd-40e9-80e6-21153c316b94" containerName="console"
Nov 02 09:38:47.253383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.253158    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809c9d21-8fdd-40e9-80e6-21153c316b94" containerName="console"
Nov 02 09:38:47.253383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.253175    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f5001d46-97a2-4947-8950-00a056377a3e" containerName="console"
Nov 02 09:38:47.253383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.253180    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5001d46-97a2-4947-8950-00a056377a3e" containerName="console"
Nov 02 09:38:47.253383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.253244    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5001d46-97a2-4947-8950-00a056377a3e" containerName="console"
Nov 02 09:38:47.253383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.253259    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="809c9d21-8fdd-40e9-80e6-21153c316b94" containerName="console"
Nov 02 09:38:47.256511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.256489    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"
Nov 02 09:38:47.270190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.270167    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-d4nmz\""
Nov 02 09:38:47.270190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.270169    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 09:38:47.272944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.272927    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 09:38:47.274445 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.274422    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"]
Nov 02 09:38:47.310725 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.310692    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n\" (UID: \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"
Nov 02 09:38:47.310898 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.310746    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n\" (UID: \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"
Nov 02 09:38:47.310898 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.310807    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcg48\" (UniqueName: \"kubernetes.io/projected/5e9392dd-369a-4558-b4d9-e7b3852bbba3-kube-api-access-wcg48\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n\" (UID: \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"
Nov 02 09:38:47.411978 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.411930    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n\" (UID: \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"
Nov 02 09:38:47.412165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.411994    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n\" (UID: \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"
Nov 02 09:38:47.412165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.412025    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcg48\" (UniqueName: \"kubernetes.io/projected/5e9392dd-369a-4558-b4d9-e7b3852bbba3-kube-api-access-wcg48\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n\" (UID: \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"
Nov 02 09:38:47.412379 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.412357    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n\" (UID: \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"
Nov 02 09:38:47.412444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.412389    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n\" (UID: \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"
Nov 02 09:38:47.425243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.425186    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcg48\" (UniqueName: \"kubernetes.io/projected/5e9392dd-369a-4558-b4d9-e7b3852bbba3-kube-api-access-wcg48\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n\" (UID: \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"
Nov 02 09:38:47.572075 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.572043    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"
Nov 02 09:38:47.848158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:47.848059    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"]
Nov 02 09:38:47.850955 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:38:47.850928    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9392dd_369a_4558_b4d9_e7b3852bbba3.slice/crio-a181d5458500fdb8b572fec7301df513236c42fa6cc3485ff354e437107807d1 WatchSource:0}: Error finding container a181d5458500fdb8b572fec7301df513236c42fa6cc3485ff354e437107807d1: Status 404 returned error can't find the container with id a181d5458500fdb8b572fec7301df513236c42fa6cc3485ff354e437107807d1
Nov 02 09:38:48.016238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:48.016188    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n" event={"ID":"5e9392dd-369a-4558-b4d9-e7b3852bbba3","Type":"ContainerStarted","Data":"a181d5458500fdb8b572fec7301df513236c42fa6cc3485ff354e437107807d1"}
Nov 02 09:38:49.917402 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:49.917354    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"]
Nov 02 09:38:49.921203 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:49.921179    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"
Nov 02 09:38:49.926014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:49.925990    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-297dn\""
Nov 02 09:38:49.931172 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:49.931097    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"]
Nov 02 09:38:50.019048 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.019006    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"]
Nov 02 09:38:50.022469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.022447    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"
Nov 02 09:38:50.026094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.026058    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-6fmjd\""
Nov 02 09:38:50.032398 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.032372    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"]
Nov 02 09:38:50.036388 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.036358    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88lb9\" (UniqueName: \"kubernetes.io/projected/093d7594-abdc-4b9a-bb57-966e6e50129e-kube-api-access-88lb9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4\" (UID: \"093d7594-abdc-4b9a-bb57-966e6e50129e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"
Nov 02 09:38:50.036552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.036428    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4\" (UID: \"093d7594-abdc-4b9a-bb57-966e6e50129e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"
Nov 02 09:38:50.036552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.036539    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4\" (UID: \"093d7594-abdc-4b9a-bb57-966e6e50129e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"
Nov 02 09:38:50.112872 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.112835    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"]
Nov 02 09:38:50.116251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.116235    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"
Nov 02 09:38:50.119853 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.119812    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-gs424\""
Nov 02 09:38:50.127580 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.127549    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"]
Nov 02 09:38:50.137271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.137242    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4\" (UID: \"093d7594-abdc-4b9a-bb57-966e6e50129e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"
Nov 02 09:38:50.137447 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.137341    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88lb9\" (UniqueName: \"kubernetes.io/projected/093d7594-abdc-4b9a-bb57-966e6e50129e-kube-api-access-88lb9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4\" (UID: \"093d7594-abdc-4b9a-bb57-966e6e50129e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"
Nov 02 09:38:50.137447 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.137408    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4\" (UID: \"093d7594-abdc-4b9a-bb57-966e6e50129e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"
Nov 02 09:38:50.137536 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.137450    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr\" (UID: \"50a250fd-1476-4dc2-b58c-87875aa93d6b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"
Nov 02 09:38:50.137536 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.137476    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr\" (UID: \"50a250fd-1476-4dc2-b58c-87875aa93d6b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"
Nov 02 09:38:50.137536 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.137500    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcsxr\" (UniqueName: \"kubernetes.io/projected/50a250fd-1476-4dc2-b58c-87875aa93d6b-kube-api-access-mcsxr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr\" (UID: \"50a250fd-1476-4dc2-b58c-87875aa93d6b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"
Nov 02 09:38:50.137744 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.137719    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4\" (UID: \"093d7594-abdc-4b9a-bb57-966e6e50129e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"
Nov 02 09:38:50.138069 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.138044    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4\" (UID: \"093d7594-abdc-4b9a-bb57-966e6e50129e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"
Nov 02 09:38:50.151027 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.151000    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88lb9\" (UniqueName: \"kubernetes.io/projected/093d7594-abdc-4b9a-bb57-966e6e50129e-kube-api-access-88lb9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4\" (UID: \"093d7594-abdc-4b9a-bb57-966e6e50129e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"
Nov 02 09:38:50.231489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.231401    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"]
Nov 02 09:38:50.235281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.235256    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"
Nov 02 09:38:50.238659 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.238631    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr\" (UID: \"50a250fd-1476-4dc2-b58c-87875aa93d6b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"
Nov 02 09:38:50.238659 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.238660    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr\" (UID: \"50a250fd-1476-4dc2-b58c-87875aa93d6b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"
Nov 02 09:38:50.238859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.238678    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcsxr\" (UniqueName: \"kubernetes.io/projected/50a250fd-1476-4dc2-b58c-87875aa93d6b-kube-api-access-mcsxr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr\" (UID: \"50a250fd-1476-4dc2-b58c-87875aa93d6b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"
Nov 02 09:38:50.238859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.238697    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2\" (UID: \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"
Nov 02 09:38:50.238859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.238738    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-875fc\" (UniqueName: \"kubernetes.io/projected/d9efa6d2-8c8c-4505-b952-299b02c45f3e-kube-api-access-875fc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2\" (UID: \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"
Nov 02 09:38:50.238859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.238781    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2\" (UID: \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"
Nov 02 09:38:50.239074 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.239050    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr\" (UID: \"50a250fd-1476-4dc2-b58c-87875aa93d6b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"
Nov 02 09:38:50.239138 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.239059    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr\" (UID: \"50a250fd-1476-4dc2-b58c-87875aa93d6b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"
Nov 02 09:38:50.239545 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.239518    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-8rdmx\""
Nov 02 09:38:50.242184 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.242162    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"
Nov 02 09:38:50.245089 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.245064    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"]
Nov 02 09:38:50.253563 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.253540    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcsxr\" (UniqueName: \"kubernetes.io/projected/50a250fd-1476-4dc2-b58c-87875aa93d6b-kube-api-access-mcsxr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr\" (UID: \"50a250fd-1476-4dc2-b58c-87875aa93d6b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"
Nov 02 09:38:50.339668 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.339632    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2\" (UID: \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"
Nov 02 09:38:50.339849 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.339681    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw\" (UID: \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"
Nov 02 09:38:50.339849 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.339730    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55z4n\" (UniqueName: \"kubernetes.io/projected/432e92b9-c705-4f37-9dbd-db584b1f5dfa-kube-api-access-55z4n\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw\" (UID: \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"
Nov 02 09:38:50.339849 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.339760    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-875fc\" (UniqueName: \"kubernetes.io/projected/d9efa6d2-8c8c-4505-b952-299b02c45f3e-kube-api-access-875fc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2\" (UID: \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"
Nov 02 09:38:50.339849 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.339795    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw\" (UID: \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"
Nov 02 09:38:50.339849 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.339831    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2\" (UID: \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"
Nov 02 09:38:50.340366 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.340334    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2\" (UID: \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"
Nov 02 09:38:50.340499 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.340411    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2\" (UID: \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"
Nov 02 09:38:50.343495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.343469    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"
Nov 02 09:38:50.353576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.353515    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-875fc\" (UniqueName: \"kubernetes.io/projected/d9efa6d2-8c8c-4505-b952-299b02c45f3e-kube-api-access-875fc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2\" (UID: \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"
Nov 02 09:38:50.436964 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.436726    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"
Nov 02 09:38:50.441028 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.440702    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw\" (UID: \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"
Nov 02 09:38:50.441028 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.440757    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55z4n\" (UniqueName: \"kubernetes.io/projected/432e92b9-c705-4f37-9dbd-db584b1f5dfa-kube-api-access-55z4n\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw\" (UID: \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"
Nov 02 09:38:50.441028 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.440818    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw\" (UID: \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"
Nov 02 09:38:50.441453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.441080    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw\" (UID: \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"
Nov 02 09:38:50.441453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.441171    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw\" (UID: \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"
Nov 02 09:38:50.453964 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.453933    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55z4n\" (UniqueName: \"kubernetes.io/projected/432e92b9-c705-4f37-9dbd-db584b1f5dfa-kube-api-access-55z4n\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw\" (UID: \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"
Nov 02 09:38:50.530445 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.530413    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"]
Nov 02 09:38:50.533130 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:38:50.533095    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093d7594_abdc_4b9a_bb57_966e6e50129e.slice/crio-826dd6732742700e3fae02105ddbb5d06ae1c56b2522f4ad2c3deea5ab188487 WatchSource:0}: Error finding container 826dd6732742700e3fae02105ddbb5d06ae1c56b2522f4ad2c3deea5ab188487: Status 404 returned error can't find the container with id 826dd6732742700e3fae02105ddbb5d06ae1c56b2522f4ad2c3deea5ab188487
Nov 02 09:38:50.555384 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.555355    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"
Nov 02 09:38:50.636850 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.636813    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"]
Nov 02 09:38:50.642495 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:38:50.642461    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a250fd_1476_4dc2_b58c_87875aa93d6b.slice/crio-dc7c7d2baef5c3d48d69e74122483fc55c1a833ce24998cd81b8ca14534b64a8 WatchSource:0}: Error finding container dc7c7d2baef5c3d48d69e74122483fc55c1a833ce24998cd81b8ca14534b64a8: Status 404 returned error can't find the container with id dc7c7d2baef5c3d48d69e74122483fc55c1a833ce24998cd81b8ca14534b64a8
Nov 02 09:38:50.733037 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.733002    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"]
Nov 02 09:38:50.839248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:50.839192    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"]
Nov 02 09:38:50.842999 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:38:50.842963    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432e92b9_c705_4f37_9dbd_db584b1f5dfa.slice/crio-e59e6fd35186b13b1a1fbd0d0b264628b1f200d5ca8c6246d852d909b4e5ba3f WatchSource:0}: Error finding container e59e6fd35186b13b1a1fbd0d0b264628b1f200d5ca8c6246d852d909b4e5ba3f: Status 404 returned error can't find the container with id e59e6fd35186b13b1a1fbd0d0b264628b1f200d5ca8c6246d852d909b4e5ba3f
Nov 02 09:38:51.030197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.030156    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2" event={"ID":"d9efa6d2-8c8c-4505-b952-299b02c45f3e","Type":"ContainerStarted","Data":"f47f984a946f8bf205660f9033a17ab7cf98886b9db3c439a97286ee3a89b335"}
Nov 02 09:38:51.032024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.031996    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4" event={"ID":"093d7594-abdc-4b9a-bb57-966e6e50129e","Type":"ContainerStarted","Data":"826dd6732742700e3fae02105ddbb5d06ae1c56b2522f4ad2c3deea5ab188487"}
Nov 02 09:38:51.033414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.033389    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr" event={"ID":"50a250fd-1476-4dc2-b58c-87875aa93d6b","Type":"ContainerStarted","Data":"dc7c7d2baef5c3d48d69e74122483fc55c1a833ce24998cd81b8ca14534b64a8"}
Nov 02 09:38:51.034736 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.034710    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw" event={"ID":"432e92b9-c705-4f37-9dbd-db584b1f5dfa","Type":"ContainerStarted","Data":"e59e6fd35186b13b1a1fbd0d0b264628b1f200d5ca8c6246d852d909b4e5ba3f"}
Nov 02 09:38:51.074657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.074617    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"]
Nov 02 09:38:51.079932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.079907    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"
Nov 02 09:38:51.088078 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.088048    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"]
Nov 02 09:38:51.088440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.088411    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2hq4x\""
Nov 02 09:38:51.147830 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.147736    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d\" (UID: \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"
Nov 02 09:38:51.147830 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.147795    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d\" (UID: \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"
Nov 02 09:38:51.148049 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.147893    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpbz7\" (UniqueName: \"kubernetes.io/projected/83bdaa43-45cb-48ae-911a-f9cece77b4ca-kube-api-access-qpbz7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d\" (UID: \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"
Nov 02 09:38:51.244776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.244737    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"]
Nov 02 09:38:51.253624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.253583    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d\" (UID: \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"
Nov 02 09:38:51.253936 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.253914    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d\" (UID: \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"
Nov 02 09:38:51.254139 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.254118    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbz7\" (UniqueName: \"kubernetes.io/projected/83bdaa43-45cb-48ae-911a-f9cece77b4ca-kube-api-access-qpbz7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d\" (UID: \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"
Nov 02 09:38:51.254266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.254210    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d\" (UID: \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"
Nov 02 09:38:51.254675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.254649    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d\" (UID: \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"
Nov 02 09:38:51.255257 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.255235    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.269366 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.268994    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"]
Nov 02 09:38:51.271734 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.269845    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 09:38:51.271734 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.269864    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 09:38:51.271734 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.269989    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 09:38:51.271734 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.270285    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b2fjp\""
Nov 02 09:38:51.280994 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.280935    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpbz7\" (UniqueName: \"kubernetes.io/projected/83bdaa43-45cb-48ae-911a-f9cece77b4ca-kube-api-access-qpbz7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d\" (UID: \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"
Nov 02 09:38:51.342277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.341583    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"]
Nov 02 09:38:51.346694 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.346068    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.358401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.355919    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c98f31-5358-46e5-825b-ab24b569f10b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.358401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.355960    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.358401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.356085    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.358401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.356171    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.358401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.356270    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.358401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.356355    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.358401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.356390    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntchs\" (UniqueName: \"kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-kube-api-access-ntchs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.358401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.356461    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.358401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.356513    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.359717 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.359693    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"]
Nov 02 09:38:51.371384 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.371340    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-6nn6l\""
Nov 02 09:38:51.400085 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.400006    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"
Nov 02 09:38:51.426370 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.426334    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"]
Nov 02 09:38:51.431448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.430763    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"
Nov 02 09:38:51.439605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.439580    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"]
Nov 02 09:38:51.457556 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.457522    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.457784 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.457759    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.457935 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.457918    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf08bc0-8270-410d-a5d4-93be687e910a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.458057 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458041    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.458166 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458153    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.458293 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458273    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.458408 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458393    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntchs\" (UniqueName: \"kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-kube-api-access-ntchs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.458526 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458512    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.458645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458632    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.458748 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458734    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.460005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458471    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.460005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458824    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.460005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458911    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.460005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458933    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf08bc0-8270-410d-a5d4-93be687e910a-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.460005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458966    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c98f31-5358-46e5-825b-ab24b569f10b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.460005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458167    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.460005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.458999    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.460005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.459029    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf08bc0-8270-410d-a5d4-93be687e910a-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.460005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.459052    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk4p6\" (UniqueName: \"kubernetes.io/projected/ccf08bc0-8270-410d-a5d4-93be687e910a-kube-api-access-tk4p6\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.460005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.459086    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.460005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.459087    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.460005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.459461    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.460005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.459957    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c98f31-5358-46e5-825b-ab24b569f10b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.469797 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.469763    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-x2wzr\""
Nov 02 09:38:51.471788 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.471764    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.473066 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.473010    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntchs\" (UniqueName: \"kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-kube-api-access-ntchs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.476425 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.476377    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.477476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.477449    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-24rkw\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.560124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.560080    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.560365 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.560137    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf08bc0-8270-410d-a5d4-93be687e910a-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.560365 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.560175    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf08bc0-8270-410d-a5d4-93be687e910a-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.560365 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.560199    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4p6\" (UniqueName: \"kubernetes.io/projected/ccf08bc0-8270-410d-a5d4-93be687e910a-kube-api-access-tk4p6\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.560365 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.560270    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.560365 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.560309    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf08bc0-8270-410d-a5d4-93be687e910a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.560624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.560370    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8\" (UID: \"b3b4c77e-436d-457e-bf06-3584adb6794f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"
Nov 02 09:38:51.560624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.560398    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.560624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.560427    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbvq\" (UniqueName: \"kubernetes.io/projected/b3b4c77e-436d-457e-bf06-3584adb6794f-kube-api-access-4mbvq\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8\" (UID: \"b3b4c77e-436d-457e-bf06-3584adb6794f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"
Nov 02 09:38:51.560624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.560457    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8\" (UID: \"b3b4c77e-436d-457e-bf06-3584adb6794f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"
Nov 02 09:38:51.560624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.560509    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.560624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.560563    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.562664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.562096    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.562664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.562425    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.562664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.562620    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.562885 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.562771    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.563469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.563398    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf08bc0-8270-410d-a5d4-93be687e910a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.565856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.565812    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf08bc0-8270-410d-a5d4-93be687e910a-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.567269 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.567202    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf08bc0-8270-410d-a5d4-93be687e910a-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.575726 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.575263    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf08bc0-8270-410d-a5d4-93be687e910a-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.575726 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.575676    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4p6\" (UniqueName: \"kubernetes.io/projected/ccf08bc0-8270-410d-a5d4-93be687e910a-kube-api-access-tk4p6\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz\" (UID: \"ccf08bc0-8270-410d-a5d4-93be687e910a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.593288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.592973    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:38:51.663126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.661935    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8\" (UID: \"b3b4c77e-436d-457e-bf06-3584adb6794f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"
Nov 02 09:38:51.663126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.661988    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbvq\" (UniqueName: \"kubernetes.io/projected/b3b4c77e-436d-457e-bf06-3584adb6794f-kube-api-access-4mbvq\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8\" (UID: \"b3b4c77e-436d-457e-bf06-3584adb6794f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"
Nov 02 09:38:51.663126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.662019    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8\" (UID: \"b3b4c77e-436d-457e-bf06-3584adb6794f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"
Nov 02 09:38:51.663126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.662504    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8\" (UID: \"b3b4c77e-436d-457e-bf06-3584adb6794f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"
Nov 02 09:38:51.663126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.662619    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8\" (UID: \"b3b4c77e-436d-457e-bf06-3584adb6794f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"
Nov 02 09:38:51.666455 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.666433    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:38:51.676717 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.676666    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbvq\" (UniqueName: \"kubernetes.io/projected/b3b4c77e-436d-457e-bf06-3584adb6794f-kube-api-access-4mbvq\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8\" (UID: \"b3b4c77e-436d-457e-bf06-3584adb6794f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"
Nov 02 09:38:51.726521 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.726483    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"]
Nov 02 09:38:51.733985 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:38:51.733874    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bdaa43_45cb_48ae_911a_f9cece77b4ca.slice/crio-f373bcd0322869590817cc806154cbb42186aeebd13449ecbf36327bd53e2840 WatchSource:0}: Error finding container f373bcd0322869590817cc806154cbb42186aeebd13449ecbf36327bd53e2840: Status 404 returned error can't find the container with id f373bcd0322869590817cc806154cbb42186aeebd13449ecbf36327bd53e2840
Nov 02 09:38:51.774036 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.773771    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"
Nov 02 09:38:51.902846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.902815    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"]
Nov 02 09:38:51.908355 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:38:51.908306    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c98f31_5358_46e5_825b_ab24b569f10b.slice/crio-601071eed90aa2afa7e66eae678788b2dc68876b57518d9ab2ce618d18c97139 WatchSource:0}: Error finding container 601071eed90aa2afa7e66eae678788b2dc68876b57518d9ab2ce618d18c97139: Status 404 returned error can't find the container with id 601071eed90aa2afa7e66eae678788b2dc68876b57518d9ab2ce618d18c97139
Nov 02 09:38:51.973143 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:51.973116    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"]
Nov 02 09:38:51.976424 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:38:51.976378    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf08bc0_8270_410d_a5d4_93be687e910a.slice/crio-663e829cfdd72333871d7448fc1a8e64d5c55af3417af8c82e30f5940d1859ab WatchSource:0}: Error finding container 663e829cfdd72333871d7448fc1a8e64d5c55af3417af8c82e30f5940d1859ab: Status 404 returned error can't find the container with id 663e829cfdd72333871d7448fc1a8e64d5c55af3417af8c82e30f5940d1859ab
Nov 02 09:38:52.042964 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:52.042873    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d" event={"ID":"83bdaa43-45cb-48ae-911a-f9cece77b4ca","Type":"ContainerStarted","Data":"f373bcd0322869590817cc806154cbb42186aeebd13449ecbf36327bd53e2840"}
Nov 02 09:38:52.045888 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:52.045796    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" event={"ID":"e5c98f31-5358-46e5-825b-ab24b569f10b","Type":"ContainerStarted","Data":"601071eed90aa2afa7e66eae678788b2dc68876b57518d9ab2ce618d18c97139"}
Nov 02 09:38:52.049937 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:52.049827    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz" event={"ID":"ccf08bc0-8270-410d-a5d4-93be687e910a","Type":"ContainerStarted","Data":"663e829cfdd72333871d7448fc1a8e64d5c55af3417af8c82e30f5940d1859ab"}
Nov 02 09:38:52.088751 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:52.088667    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"]
Nov 02 09:38:52.092584 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:38:52.092541    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b4c77e_436d_457e_bf06_3584adb6794f.slice/crio-1b9e5141299ddb8cbabf479fbedd61f33308743dac41369bd1f00047087e3aee WatchSource:0}: Error finding container 1b9e5141299ddb8cbabf479fbedd61f33308743dac41369bd1f00047087e3aee: Status 404 returned error can't find the container with id 1b9e5141299ddb8cbabf479fbedd61f33308743dac41369bd1f00047087e3aee
Nov 02 09:38:53.079883 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:38:53.079820    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8" event={"ID":"b3b4c77e-436d-457e-bf06-3584adb6794f","Type":"ContainerStarted","Data":"1b9e5141299ddb8cbabf479fbedd61f33308743dac41369bd1f00047087e3aee"}
Nov 02 09:39:01.006640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:01.006590    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:39:01.007059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:01.006683    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:39:01.007059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:01.006732    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:39:01.141576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:01.141536    4449 generic.go:356] "Generic (PLEG): container finished" podID="5e9392dd-369a-4558-b4d9-e7b3852bbba3" containerID="befad33c2daaf5358674e6377d9b1abeed0e12b5ab1606c02cce5aa132920e3a" exitCode=0
Nov 02 09:39:01.141770 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:01.141691    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n" event={"ID":"5e9392dd-369a-4558-b4d9-e7b3852bbba3","Type":"ContainerDied","Data":"befad33c2daaf5358674e6377d9b1abeed0e12b5ab1606c02cce5aa132920e3a"}
Nov 02 09:39:01.149029 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:01.148955    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz" event={"ID":"ccf08bc0-8270-410d-a5d4-93be687e910a","Type":"ContainerStarted","Data":"f8cdfd20a5f182c0a5aada0700430721395b70d4262195ebfa2038d20cb530b3"}
Nov 02 09:39:01.189799 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:01.189586    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz" podStartSLOduration=1.166788242 podStartE2EDuration="10.189568236s" totalImagesPullingTime="9.022779994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:01.187644746 +0000 UTC m=+218.125931523" watchObservedRunningTime="2026-11-02 09:39:01.189568236 +0000 UTC m=+218.127855013"
Nov 02 09:39:01.667237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:01.667195    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:39:02.154831 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.154754    4449 generic.go:356] "Generic (PLEG): container finished" podID="093d7594-abdc-4b9a-bb57-966e6e50129e" containerID="17c70aaa4641db735a0cb1523e067ff6cf1f0e6481c19a874f93de629210da18" exitCode=0
Nov 02 09:39:02.155272 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.154828    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4" event={"ID":"093d7594-abdc-4b9a-bb57-966e6e50129e","Type":"ContainerDied","Data":"17c70aaa4641db735a0cb1523e067ff6cf1f0e6481c19a874f93de629210da18"}
Nov 02 09:39:02.156818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.156781    4449 generic.go:356] "Generic (PLEG): container finished" podID="83bdaa43-45cb-48ae-911a-f9cece77b4ca" containerID="672729b31f6595336ef629ee89270c5a30bfdce60c4aab9ea9b8492b50902090" exitCode=0
Nov 02 09:39:02.156929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.156827    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d" event={"ID":"83bdaa43-45cb-48ae-911a-f9cece77b4ca","Type":"ContainerDied","Data":"672729b31f6595336ef629ee89270c5a30bfdce60c4aab9ea9b8492b50902090"}
Nov 02 09:39:02.158667 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.158561    4449 generic.go:356] "Generic (PLEG): container finished" podID="50a250fd-1476-4dc2-b58c-87875aa93d6b" containerID="bd0bbfaf7a46ee20361c91626fee6614882c28e8a8888c245658ed35b5cd1402" exitCode=0
Nov 02 09:39:02.158667 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.158626    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr" event={"ID":"50a250fd-1476-4dc2-b58c-87875aa93d6b","Type":"ContainerDied","Data":"bd0bbfaf7a46ee20361c91626fee6614882c28e8a8888c245658ed35b5cd1402"}
Nov 02 09:39:02.160424 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.160403    4449 generic.go:356] "Generic (PLEG): container finished" podID="432e92b9-c705-4f37-9dbd-db584b1f5dfa" containerID="c2a1607164a7c6085452f612bb00c5595a4230f08f8796233cb4fe8b1bae3fe8" exitCode=0
Nov 02 09:39:02.160491 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.160449    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw" event={"ID":"432e92b9-c705-4f37-9dbd-db584b1f5dfa","Type":"ContainerDied","Data":"c2a1607164a7c6085452f612bb00c5595a4230f08f8796233cb4fe8b1bae3fe8"}
Nov 02 09:39:02.162069 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.162049    4449 generic.go:356] "Generic (PLEG): container finished" podID="b3b4c77e-436d-457e-bf06-3584adb6794f" containerID="244dd09f1c8dea6490d48b9c91181a929d0a60ef826182f8ccfcbc3f563f8ac1" exitCode=0
Nov 02 09:39:02.162172 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.162072    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8" event={"ID":"b3b4c77e-436d-457e-bf06-3584adb6794f","Type":"ContainerDied","Data":"244dd09f1c8dea6490d48b9c91181a929d0a60ef826182f8ccfcbc3f563f8ac1"}
Nov 02 09:39:02.474417 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.474380    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:39:02.474490 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.474465    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:39:02.474526 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.474495    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:39:02.672330 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:02.672280    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:39:03.170230 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:03.170186    4449 generic.go:356] "Generic (PLEG): container finished" podID="d9efa6d2-8c8c-4505-b952-299b02c45f3e" containerID="eb34f91dec8e0011e76a94961ab306245745aa8762be17d285c07c8a8144b09b" exitCode=0
Nov 02 09:39:03.170636 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:03.170318    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2" event={"ID":"d9efa6d2-8c8c-4505-b952-299b02c45f3e","Type":"ContainerDied","Data":"eb34f91dec8e0011e76a94961ab306245745aa8762be17d285c07c8a8144b09b"}
Nov 02 09:39:03.173885 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:03.173783    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" event={"ID":"e5c98f31-5358-46e5-825b-ab24b569f10b","Type":"ContainerStarted","Data":"9b651472f6f70fe339a759e86aa53cbf0eafa4727ba38e18089fa52e7a1a6b0f"}
Nov 02 09:39:03.176708 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:03.176462    4449 generic.go:356] "Generic (PLEG): container finished" podID="b3b4c77e-436d-457e-bf06-3584adb6794f" containerID="67b9a39c8fc1c639613b6ad9a5fc33c9875d26c345d451278e77a0ca6de01286" exitCode=0
Nov 02 09:39:03.177954 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:03.177933    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8" event={"ID":"b3b4c77e-436d-457e-bf06-3584adb6794f","Type":"ContainerDied","Data":"67b9a39c8fc1c639613b6ad9a5fc33c9875d26c345d451278e77a0ca6de01286"}
Nov 02 09:39:03.178032 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:03.177982    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:39:03.178958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:03.178941    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz"
Nov 02 09:39:03.279273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:03.279098    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" podStartSLOduration=1.716716748 podStartE2EDuration="12.279078369s" totalImagesPullingTime="10.562361621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:03.272568923 +0000 UTC m=+220.210855761" watchObservedRunningTime="2026-11-02 09:39:03.279078369 +0000 UTC m=+220.217365144"
Nov 02 09:39:03.593237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:03.593177    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:39:03.599638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:03.599608    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:39:04.187125 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:04.186015    4449 generic.go:356] "Generic (PLEG): container finished" podID="093d7594-abdc-4b9a-bb57-966e6e50129e" containerID="3e354e9a450c40d7de48c61b94fe031eab020dd8939cc3193c5567231856c25e" exitCode=0
Nov 02 09:39:04.187125 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:04.186116    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4" event={"ID":"093d7594-abdc-4b9a-bb57-966e6e50129e","Type":"ContainerDied","Data":"3e354e9a450c40d7de48c61b94fe031eab020dd8939cc3193c5567231856c25e"}
Nov 02 09:39:04.192239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:04.192147    4449 generic.go:356] "Generic (PLEG): container finished" podID="50a250fd-1476-4dc2-b58c-87875aa93d6b" containerID="4b77935b488e7b5b9f3ef4e9db5113a5dad820c58bf441a567ebfec93e2eea36" exitCode=0
Nov 02 09:39:04.192385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:04.192284    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr" event={"ID":"50a250fd-1476-4dc2-b58c-87875aa93d6b","Type":"ContainerDied","Data":"4b77935b488e7b5b9f3ef4e9db5113a5dad820c58bf441a567ebfec93e2eea36"}
Nov 02 09:39:04.197126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:04.197102    4449 generic.go:356] "Generic (PLEG): container finished" podID="432e92b9-c705-4f37-9dbd-db584b1f5dfa" containerID="8e09c12e80defa00aef50536c5bbb3b45831e62a55ea8432eae817e9f9476d7a" exitCode=0
Nov 02 09:39:04.198194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:04.198165    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw" event={"ID":"432e92b9-c705-4f37-9dbd-db584b1f5dfa","Type":"ContainerDied","Data":"8e09c12e80defa00aef50536c5bbb3b45831e62a55ea8432eae817e9f9476d7a"}
Nov 02 09:39:04.198307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:04.198249    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:39:04.199203 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:04.199176    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:39:05.209240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:05.209181    4449 generic.go:356] "Generic (PLEG): container finished" podID="d9efa6d2-8c8c-4505-b952-299b02c45f3e" containerID="108491e1951d9b1e6fabd6882d2034f7a522172acbf9ec333904168bb21c518b" exitCode=0
Nov 02 09:39:05.210638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:05.210614    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2" event={"ID":"d9efa6d2-8c8c-4505-b952-299b02c45f3e","Type":"ContainerDied","Data":"108491e1951d9b1e6fabd6882d2034f7a522172acbf9ec333904168bb21c518b"}
Nov 02 09:39:06.219843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:06.219807    4449 generic.go:356] "Generic (PLEG): container finished" podID="5e9392dd-369a-4558-b4d9-e7b3852bbba3" containerID="64fd7f3aa2d6634b1f56507a558c3121e29d7771a0d248c8bbdac1781d4e0255" exitCode=0
Nov 02 09:39:06.220342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:06.219917    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n" event={"ID":"5e9392dd-369a-4558-b4d9-e7b3852bbba3","Type":"ContainerDied","Data":"64fd7f3aa2d6634b1f56507a558c3121e29d7771a0d248c8bbdac1781d4e0255"}
Nov 02 09:39:06.228146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:06.226790    4449 generic.go:356] "Generic (PLEG): container finished" podID="83bdaa43-45cb-48ae-911a-f9cece77b4ca" containerID="9e8415bd3dba788b4921a429247dea89cfe68d766b1885ebad2ae5bef7cdfd93" exitCode=0
Nov 02 09:39:06.228146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:06.227943    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d" event={"ID":"83bdaa43-45cb-48ae-911a-f9cece77b4ca","Type":"ContainerDied","Data":"9e8415bd3dba788b4921a429247dea89cfe68d766b1885ebad2ae5bef7cdfd93"}
Nov 02 09:39:06.579070 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:39:06.579027    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:39:18.346827 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:18.346794    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr" event={"ID":"50a250fd-1476-4dc2-b58c-87875aa93d6b","Type":"ContainerStarted","Data":"eda71a18f273f3dc8d67e6b2ad0ef76510c8eff5bc74825658c057d43d55267b"}
Nov 02 09:39:18.350051 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:18.350017    4449 generic.go:356] "Generic (PLEG): container finished" podID="b3b4c77e-436d-457e-bf06-3584adb6794f" containerID="9b27766130ed77e7bc9f443c4a0e34a638597fe9b0568b1001f73fccb97d2812" exitCode=0
Nov 02 09:39:18.350184 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:18.350124    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8" event={"ID":"b3b4c77e-436d-457e-bf06-3584adb6794f","Type":"ContainerDied","Data":"9b27766130ed77e7bc9f443c4a0e34a638597fe9b0568b1001f73fccb97d2812"}
Nov 02 09:39:18.390266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:18.390188    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr" podStartSLOduration=2.153451076 podStartE2EDuration="28.390174877s" totalImagesPullingTime="26.236723801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:38:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:18.369087998 +0000 UTC m=+235.307374777" watchObservedRunningTime="2026-11-02 09:39:18.390174877 +0000 UTC m=+235.328461653"
Nov 02 09:39:19.359275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.359000    4449 generic.go:356] "Generic (PLEG): container finished" podID="432e92b9-c705-4f37-9dbd-db584b1f5dfa" containerID="4b18b59e8a279315b60e21903c335c2731b1b21a37eb54162d2b1e2bfa570f05" exitCode=0
Nov 02 09:39:19.359275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.359177    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw" event={"ID":"432e92b9-c705-4f37-9dbd-db584b1f5dfa","Type":"ContainerDied","Data":"4b18b59e8a279315b60e21903c335c2731b1b21a37eb54162d2b1e2bfa570f05"}
Nov 02 09:39:19.362426 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.362290    4449 generic.go:356] "Generic (PLEG): container finished" podID="093d7594-abdc-4b9a-bb57-966e6e50129e" containerID="216f04bba786b653f12c5fb2a3c632441cd9fd6070110a8698f7e892c4065bef" exitCode=0
Nov 02 09:39:19.362426 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.362381    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4" event={"ID":"093d7594-abdc-4b9a-bb57-966e6e50129e","Type":"ContainerDied","Data":"216f04bba786b653f12c5fb2a3c632441cd9fd6070110a8698f7e892c4065bef"}
Nov 02 09:39:19.365156 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.365132    4449 generic.go:356] "Generic (PLEG): container finished" podID="50a250fd-1476-4dc2-b58c-87875aa93d6b" containerID="eda71a18f273f3dc8d67e6b2ad0ef76510c8eff5bc74825658c057d43d55267b" exitCode=0
Nov 02 09:39:19.365371 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.365349    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr" event={"ID":"50a250fd-1476-4dc2-b58c-87875aa93d6b","Type":"ContainerDied","Data":"eda71a18f273f3dc8d67e6b2ad0ef76510c8eff5bc74825658c057d43d55267b"}
Nov 02 09:39:19.568028 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.567996    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"
Nov 02 09:39:19.656128 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.656042    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-util\" (UniqueName: \"kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-util\") pod \"b3b4c77e-436d-457e-bf06-3584adb6794f\" (UID: \"b3b4c77e-436d-457e-bf06-3584adb6794f\") "
Nov 02 09:39:19.656335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.656136    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-bundle\") pod \"b3b4c77e-436d-457e-bf06-3584adb6794f\" (UID: \"b3b4c77e-436d-457e-bf06-3584adb6794f\") "
Nov 02 09:39:19.656335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.656160    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b4c77e-436d-457e-bf06-3584adb6794f-kube-api-access-4mbvq\" (UniqueName: \"kubernetes.io/projected/b3b4c77e-436d-457e-bf06-3584adb6794f-kube-api-access-4mbvq\") pod \"b3b4c77e-436d-457e-bf06-3584adb6794f\" (UID: \"b3b4c77e-436d-457e-bf06-3584adb6794f\") "
Nov 02 09:39:19.658381 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.658328    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-bundle" pod "b3b4c77e-436d-457e-bf06-3584adb6794f" (UID: "b3b4c77e-436d-457e-bf06-3584adb6794f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:19.659343 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.659317    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b4c77e-436d-457e-bf06-3584adb6794f-kube-api-access-4mbvq" pod "b3b4c77e-436d-457e-bf06-3584adb6794f" (UID: "b3b4c77e-436d-457e-bf06-3584adb6794f"). InnerVolumeSpecName "kube-api-access-4mbvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:39:19.662758 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.662725    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-util" pod "b3b4c77e-436d-457e-bf06-3584adb6794f" (UID: "b3b4c77e-436d-457e-bf06-3584adb6794f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:19.757231 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.757182    4449 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-util\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:19.757231 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.757228    4449 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3b4c77e-436d-457e-bf06-3584adb6794f-bundle\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:19.757231 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:19.757242    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mbvq\" (UniqueName: \"kubernetes.io/projected/b3b4c77e-436d-457e-bf06-3584adb6794f-kube-api-access-4mbvq\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:20.371357 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.371318    4449 generic.go:356] "Generic (PLEG): container finished" podID="d9efa6d2-8c8c-4505-b952-299b02c45f3e" containerID="51bb43046fb8cbebb5e50426cca19302a22ea6a1745e8dce473c4a057edbd9c1" exitCode=0
Nov 02 09:39:20.371798 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.371400    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2" event={"ID":"d9efa6d2-8c8c-4505-b952-299b02c45f3e","Type":"ContainerDied","Data":"51bb43046fb8cbebb5e50426cca19302a22ea6a1745e8dce473c4a057edbd9c1"}
Nov 02 09:39:20.373408 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.373376    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8" event={"ID":"b3b4c77e-436d-457e-bf06-3584adb6794f","Type":"ContainerDied","Data":"1b9e5141299ddb8cbabf479fbedd61f33308743dac41369bd1f00047087e3aee"}
Nov 02 09:39:20.373499 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.373408    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71psnl8"
Nov 02 09:39:20.373499 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.373426    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b9e5141299ddb8cbabf479fbedd61f33308743dac41369bd1f00047087e3aee"
Nov 02 09:39:20.529346 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.529202    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"
Nov 02 09:39:20.563633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.563595    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093d7594-abdc-4b9a-bb57-966e6e50129e-kube-api-access-88lb9\" (UniqueName: \"kubernetes.io/projected/093d7594-abdc-4b9a-bb57-966e6e50129e-kube-api-access-88lb9\") pod \"093d7594-abdc-4b9a-bb57-966e6e50129e\" (UID: \"093d7594-abdc-4b9a-bb57-966e6e50129e\") "
Nov 02 09:39:20.563819 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.563698    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-bundle\") pod \"093d7594-abdc-4b9a-bb57-966e6e50129e\" (UID: \"093d7594-abdc-4b9a-bb57-966e6e50129e\") "
Nov 02 09:39:20.563819 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.563748    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-util\" (UniqueName: \"kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-util\") pod \"093d7594-abdc-4b9a-bb57-966e6e50129e\" (UID: \"093d7594-abdc-4b9a-bb57-966e6e50129e\") "
Nov 02 09:39:20.564167 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.564138    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"
Nov 02 09:39:20.564566 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.564525    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-bundle" pod "093d7594-abdc-4b9a-bb57-966e6e50129e" (UID: "093d7594-abdc-4b9a-bb57-966e6e50129e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:20.566050 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.566018    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093d7594-abdc-4b9a-bb57-966e6e50129e-kube-api-access-88lb9" pod "093d7594-abdc-4b9a-bb57-966e6e50129e" (UID: "093d7594-abdc-4b9a-bb57-966e6e50129e"). InnerVolumeSpecName "kube-api-access-88lb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:39:20.570159 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.570131    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-util" pod "093d7594-abdc-4b9a-bb57-966e6e50129e" (UID: "093d7594-abdc-4b9a-bb57-966e6e50129e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:20.591047 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.591021    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"
Nov 02 09:39:20.664561 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.664532    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-util\" (UniqueName: \"kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-util\") pod \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\" (UID: \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\") "
Nov 02 09:39:20.664709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.664578    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-util\" (UniqueName: \"kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-util\") pod \"50a250fd-1476-4dc2-b58c-87875aa93d6b\" (UID: \"50a250fd-1476-4dc2-b58c-87875aa93d6b\") "
Nov 02 09:39:20.664709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.664598    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432e92b9-c705-4f37-9dbd-db584b1f5dfa-kube-api-access-55z4n\" (UniqueName: \"kubernetes.io/projected/432e92b9-c705-4f37-9dbd-db584b1f5dfa-kube-api-access-55z4n\") pod \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\" (UID: \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\") "
Nov 02 09:39:20.664709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.664670    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a250fd-1476-4dc2-b58c-87875aa93d6b-kube-api-access-mcsxr\" (UniqueName: \"kubernetes.io/projected/50a250fd-1476-4dc2-b58c-87875aa93d6b-kube-api-access-mcsxr\") pod \"50a250fd-1476-4dc2-b58c-87875aa93d6b\" (UID: \"50a250fd-1476-4dc2-b58c-87875aa93d6b\") "
Nov 02 09:39:20.664709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.664692    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-bundle\") pod \"50a250fd-1476-4dc2-b58c-87875aa93d6b\" (UID: \"50a250fd-1476-4dc2-b58c-87875aa93d6b\") "
Nov 02 09:39:20.664924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.664719    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-bundle\") pod \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\" (UID: \"432e92b9-c705-4f37-9dbd-db584b1f5dfa\") "
Nov 02 09:39:20.664978 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.664947    4449 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-util\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:20.664978 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.664968    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88lb9\" (UniqueName: \"kubernetes.io/projected/093d7594-abdc-4b9a-bb57-966e6e50129e-kube-api-access-88lb9\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:20.665083 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.664986    4449 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/093d7594-abdc-4b9a-bb57-966e6e50129e-bundle\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:20.665639 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.665602    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-bundle" pod "50a250fd-1476-4dc2-b58c-87875aa93d6b" (UID: "50a250fd-1476-4dc2-b58c-87875aa93d6b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:20.665874 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.665846    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-bundle" pod "432e92b9-c705-4f37-9dbd-db584b1f5dfa" (UID: "432e92b9-c705-4f37-9dbd-db584b1f5dfa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:20.666921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.666890    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432e92b9-c705-4f37-9dbd-db584b1f5dfa-kube-api-access-55z4n" pod "432e92b9-c705-4f37-9dbd-db584b1f5dfa" (UID: "432e92b9-c705-4f37-9dbd-db584b1f5dfa"). InnerVolumeSpecName "kube-api-access-55z4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:39:20.667177 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.667157    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a250fd-1476-4dc2-b58c-87875aa93d6b-kube-api-access-mcsxr" pod "50a250fd-1476-4dc2-b58c-87875aa93d6b" (UID: "50a250fd-1476-4dc2-b58c-87875aa93d6b"). InnerVolumeSpecName "kube-api-access-mcsxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:39:20.671677 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.671658    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-util" pod "432e92b9-c705-4f37-9dbd-db584b1f5dfa" (UID: "432e92b9-c705-4f37-9dbd-db584b1f5dfa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:20.672328 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.672304    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-util" pod "50a250fd-1476-4dc2-b58c-87875aa93d6b" (UID: "50a250fd-1476-4dc2-b58c-87875aa93d6b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:20.766960 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.766929    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcsxr\" (UniqueName: \"kubernetes.io/projected/50a250fd-1476-4dc2-b58c-87875aa93d6b-kube-api-access-mcsxr\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:20.767061 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.766964    4449 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-bundle\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:20.767061 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.766980    4449 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-bundle\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:20.767061 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.766993    4449 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/432e92b9-c705-4f37-9dbd-db584b1f5dfa-util\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:20.767061 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.767005    4449 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a250fd-1476-4dc2-b58c-87875aa93d6b-util\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:20.767061 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:20.767019    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55z4n\" (UniqueName: \"kubernetes.io/projected/432e92b9-c705-4f37-9dbd-db584b1f5dfa-kube-api-access-55z4n\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:21.379666 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.379629    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr"
Nov 02 09:39:21.379666 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.379633    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cshpmr" event={"ID":"50a250fd-1476-4dc2-b58c-87875aa93d6b","Type":"ContainerDied","Data":"dc7c7d2baef5c3d48d69e74122483fc55c1a833ce24998cd81b8ca14534b64a8"}
Nov 02 09:39:21.380245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.379684    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc7c7d2baef5c3d48d69e74122483fc55c1a833ce24998cd81b8ca14534b64a8"
Nov 02 09:39:21.381820 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.381799    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw"
Nov 02 09:39:21.381980 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.381885    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab89vw" event={"ID":"432e92b9-c705-4f37-9dbd-db584b1f5dfa","Type":"ContainerDied","Data":"e59e6fd35186b13b1a1fbd0d0b264628b1f200d5ca8c6246d852d909b4e5ba3f"}
Nov 02 09:39:21.381980 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.381926    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e59e6fd35186b13b1a1fbd0d0b264628b1f200d5ca8c6246d852d909b4e5ba3f"
Nov 02 09:39:21.384552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.384532    4449 generic.go:356] "Generic (PLEG): container finished" podID="5e9392dd-369a-4558-b4d9-e7b3852bbba3" containerID="70b11449a7374df80affdc242bd3ab8195d74b50efdd26f94240bcf1829aab84" exitCode=0
Nov 02 09:39:21.384654 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.384606    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n" event={"ID":"5e9392dd-369a-4558-b4d9-e7b3852bbba3","Type":"ContainerDied","Data":"70b11449a7374df80affdc242bd3ab8195d74b50efdd26f94240bcf1829aab84"}
Nov 02 09:39:21.386481 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.386458    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4" event={"ID":"093d7594-abdc-4b9a-bb57-966e6e50129e","Type":"ContainerDied","Data":"826dd6732742700e3fae02105ddbb5d06ae1c56b2522f4ad2c3deea5ab188487"}
Nov 02 09:39:21.386586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.386483    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74bhr4"
Nov 02 09:39:21.386586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.386494    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="826dd6732742700e3fae02105ddbb5d06ae1c56b2522f4ad2c3deea5ab188487"
Nov 02 09:39:21.388993 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.388970    4449 generic.go:356] "Generic (PLEG): container finished" podID="83bdaa43-45cb-48ae-911a-f9cece77b4ca" containerID="d32ae9705f4c07a0ffeefa0caee6ebad72581b2286eecc80499da2d568df1827" exitCode=0
Nov 02 09:39:21.389094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.388987    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d" event={"ID":"83bdaa43-45cb-48ae-911a-f9cece77b4ca","Type":"ContainerDied","Data":"d32ae9705f4c07a0ffeefa0caee6ebad72581b2286eecc80499da2d568df1827"}
Nov 02 09:39:21.512433 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.512402    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"
Nov 02 09:39:21.574639 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.574603    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-util\" (UniqueName: \"kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-util\") pod \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\" (UID: \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\") "
Nov 02 09:39:21.574639 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.574645    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9efa6d2-8c8c-4505-b952-299b02c45f3e-kube-api-access-875fc\" (UniqueName: \"kubernetes.io/projected/d9efa6d2-8c8c-4505-b952-299b02c45f3e-kube-api-access-875fc\") pod \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\" (UID: \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\") "
Nov 02 09:39:21.574930 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.574726    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-bundle\") pod \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\" (UID: \"d9efa6d2-8c8c-4505-b952-299b02c45f3e\") "
Nov 02 09:39:21.575281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.575250    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-bundle" pod "d9efa6d2-8c8c-4505-b952-299b02c45f3e" (UID: "d9efa6d2-8c8c-4505-b952-299b02c45f3e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:21.576936 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.576914    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9efa6d2-8c8c-4505-b952-299b02c45f3e-kube-api-access-875fc" pod "d9efa6d2-8c8c-4505-b952-299b02c45f3e" (UID: "d9efa6d2-8c8c-4505-b952-299b02c45f3e"). InnerVolumeSpecName "kube-api-access-875fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:39:21.580681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.580653    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-util" pod "d9efa6d2-8c8c-4505-b952-299b02c45f3e" (UID: "d9efa6d2-8c8c-4505-b952-299b02c45f3e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:21.675948 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.675896    4449 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-util\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:21.675948 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.675944    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-875fc\" (UniqueName: \"kubernetes.io/projected/d9efa6d2-8c8c-4505-b952-299b02c45f3e-kube-api-access-875fc\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:21.675948 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:21.675953    4449 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9efa6d2-8c8c-4505-b952-299b02c45f3e-bundle\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:22.396564 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.396526    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2" event={"ID":"d9efa6d2-8c8c-4505-b952-299b02c45f3e","Type":"ContainerDied","Data":"f47f984a946f8bf205660f9033a17ab7cf98886b9db3c439a97286ee3a89b335"}
Nov 02 09:39:22.396564 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.396557    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cq7h2"
Nov 02 09:39:22.397017 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.396576    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f47f984a946f8bf205660f9033a17ab7cf98886b9db3c439a97286ee3a89b335"
Nov 02 09:39:22.552112 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.552089    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"
Nov 02 09:39:22.557564 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.557540    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"
Nov 02 09:39:22.584232 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.584183    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9392dd-369a-4558-b4d9-e7b3852bbba3-kube-api-access-wcg48\" (UniqueName: \"kubernetes.io/projected/5e9392dd-369a-4558-b4d9-e7b3852bbba3-kube-api-access-wcg48\") pod \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\" (UID: \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\") "
Nov 02 09:39:22.584399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.584248    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-util\" (UniqueName: \"kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-util\") pod \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\" (UID: \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\") "
Nov 02 09:39:22.584399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.584344    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-bundle\") pod \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\" (UID: \"5e9392dd-369a-4558-b4d9-e7b3852bbba3\") "
Nov 02 09:39:22.584698 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.584678    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-bundle" pod "5e9392dd-369a-4558-b4d9-e7b3852bbba3" (UID: "5e9392dd-369a-4558-b4d9-e7b3852bbba3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:22.586625 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.586596    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9392dd-369a-4558-b4d9-e7b3852bbba3-kube-api-access-wcg48" pod "5e9392dd-369a-4558-b4d9-e7b3852bbba3" (UID: "5e9392dd-369a-4558-b4d9-e7b3852bbba3"). InnerVolumeSpecName "kube-api-access-wcg48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:39:22.591850 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.591824    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-util" pod "5e9392dd-369a-4558-b4d9-e7b3852bbba3" (UID: "5e9392dd-369a-4558-b4d9-e7b3852bbba3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:22.685675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.685590    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-bundle\") pod \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\" (UID: \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\") "
Nov 02 09:39:22.685675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.685650    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-util\" (UniqueName: \"kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-util\") pod \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\" (UID: \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\") "
Nov 02 09:39:22.685854 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.685686    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83bdaa43-45cb-48ae-911a-f9cece77b4ca-kube-api-access-qpbz7\" (UniqueName: \"kubernetes.io/projected/83bdaa43-45cb-48ae-911a-f9cece77b4ca-kube-api-access-qpbz7\") pod \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\" (UID: \"83bdaa43-45cb-48ae-911a-f9cece77b4ca\") "
Nov 02 09:39:22.692675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.686279    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcg48\" (UniqueName: \"kubernetes.io/projected/5e9392dd-369a-4558-b4d9-e7b3852bbba3-kube-api-access-wcg48\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:22.692675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.686330    4449 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-util\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:22.692675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.686345    4449 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e9392dd-369a-4558-b4d9-e7b3852bbba3-bundle\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:22.692675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.686906    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-bundle" pod "83bdaa43-45cb-48ae-911a-f9cece77b4ca" (UID: "83bdaa43-45cb-48ae-911a-f9cece77b4ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:22.692998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.692878    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bdaa43-45cb-48ae-911a-f9cece77b4ca-kube-api-access-qpbz7" pod "83bdaa43-45cb-48ae-911a-f9cece77b4ca" (UID: "83bdaa43-45cb-48ae-911a-f9cece77b4ca"). InnerVolumeSpecName "kube-api-access-qpbz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:39:22.693624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.693589    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-util" pod "83bdaa43-45cb-48ae-911a-f9cece77b4ca" (UID: "83bdaa43-45cb-48ae-911a-f9cece77b4ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:22.786712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.786670    4449 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-util\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:22.786885 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.786728    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpbz7\" (UniqueName: \"kubernetes.io/projected/83bdaa43-45cb-48ae-911a-f9cece77b4ca-kube-api-access-qpbz7\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:22.786885 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:22.786739    4449 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bdaa43-45cb-48ae-911a-f9cece77b4ca-bundle\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:23.402847 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:23.402815    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d"
Nov 02 09:39:23.403338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:23.402820    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csfg7d" event={"ID":"83bdaa43-45cb-48ae-911a-f9cece77b4ca","Type":"ContainerDied","Data":"f373bcd0322869590817cc806154cbb42186aeebd13449ecbf36327bd53e2840"}
Nov 02 09:39:23.403338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:23.402937    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f373bcd0322869590817cc806154cbb42186aeebd13449ecbf36327bd53e2840"
Nov 02 09:39:23.405102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:23.405077    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n" event={"ID":"5e9392dd-369a-4558-b4d9-e7b3852bbba3","Type":"ContainerDied","Data":"a181d5458500fdb8b572fec7301df513236c42fa6cc3485ff354e437107807d1"}
Nov 02 09:39:23.405102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:23.405114    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a181d5458500fdb8b572fec7301df513236c42fa6cc3485ff354e437107807d1"
Nov 02 09:39:23.405298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:23.405132    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644crw8n"
Nov 02 09:39:24.415085 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415046    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"]
Nov 02 09:39:24.415638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415537    4449 cpu_manager.go:414] "containerMap: removing container" podUID="d9efa6d2-8c8c-4505-b952-299b02c45f3e" containerName="util"
Nov 02 09:39:24.415638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415555    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9efa6d2-8c8c-4505-b952-299b02c45f3e" containerName="util"
Nov 02 09:39:24.415638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415568    4449 cpu_manager.go:414] "containerMap: removing container" podUID="83bdaa43-45cb-48ae-911a-f9cece77b4ca" containerName="pull"
Nov 02 09:39:24.415638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415575    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bdaa43-45cb-48ae-911a-f9cece77b4ca" containerName="pull"
Nov 02 09:39:24.415638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415590    4449 cpu_manager.go:414] "containerMap: removing container" podUID="432e92b9-c705-4f37-9dbd-db584b1f5dfa" containerName="extract"
Nov 02 09:39:24.415638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415597    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432e92b9-c705-4f37-9dbd-db584b1f5dfa" containerName="extract"
Nov 02 09:39:24.415638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415605    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5e9392dd-369a-4558-b4d9-e7b3852bbba3" containerName="extract"
Nov 02 09:39:24.415638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415611    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9392dd-369a-4558-b4d9-e7b3852bbba3" containerName="extract"
Nov 02 09:39:24.415638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415625    4449 cpu_manager.go:414] "containerMap: removing container" podUID="50a250fd-1476-4dc2-b58c-87875aa93d6b" containerName="extract"
Nov 02 09:39:24.415638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415633    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a250fd-1476-4dc2-b58c-87875aa93d6b" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415646    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b3b4c77e-436d-457e-bf06-3584adb6794f" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415654    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b4c77e-436d-457e-bf06-3584adb6794f" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415663    4449 cpu_manager.go:414] "containerMap: removing container" podUID="d9efa6d2-8c8c-4505-b952-299b02c45f3e" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415670    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9efa6d2-8c8c-4505-b952-299b02c45f3e" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415680    4449 cpu_manager.go:414] "containerMap: removing container" podUID="432e92b9-c705-4f37-9dbd-db584b1f5dfa" containerName="pull"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415687    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432e92b9-c705-4f37-9dbd-db584b1f5dfa" containerName="pull"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415699    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5e9392dd-369a-4558-b4d9-e7b3852bbba3" containerName="pull"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415706    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9392dd-369a-4558-b4d9-e7b3852bbba3" containerName="pull"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415715    4449 cpu_manager.go:414] "containerMap: removing container" podUID="093d7594-abdc-4b9a-bb57-966e6e50129e" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415721    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093d7594-abdc-4b9a-bb57-966e6e50129e" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415734    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5e9392dd-369a-4558-b4d9-e7b3852bbba3" containerName="util"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415741    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9392dd-369a-4558-b4d9-e7b3852bbba3" containerName="util"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415750    4449 cpu_manager.go:414] "containerMap: removing container" podUID="d9efa6d2-8c8c-4505-b952-299b02c45f3e" containerName="pull"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415756    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9efa6d2-8c8c-4505-b952-299b02c45f3e" containerName="pull"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415769    4449 cpu_manager.go:414] "containerMap: removing container" podUID="432e92b9-c705-4f37-9dbd-db584b1f5dfa" containerName="util"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415777    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432e92b9-c705-4f37-9dbd-db584b1f5dfa" containerName="util"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415793    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b3b4c77e-436d-457e-bf06-3584adb6794f" containerName="pull"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415799    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b4c77e-436d-457e-bf06-3584adb6794f" containerName="pull"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415812    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b3b4c77e-436d-457e-bf06-3584adb6794f" containerName="util"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415819    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b4c77e-436d-457e-bf06-3584adb6794f" containerName="util"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415829    4449 cpu_manager.go:414] "containerMap: removing container" podUID="50a250fd-1476-4dc2-b58c-87875aa93d6b" containerName="util"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415838    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a250fd-1476-4dc2-b58c-87875aa93d6b" containerName="util"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415849    4449 cpu_manager.go:414] "containerMap: removing container" podUID="83bdaa43-45cb-48ae-911a-f9cece77b4ca" containerName="util"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415855    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bdaa43-45cb-48ae-911a-f9cece77b4ca" containerName="util"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415866    4449 cpu_manager.go:414] "containerMap: removing container" podUID="093d7594-abdc-4b9a-bb57-966e6e50129e" containerName="pull"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415873    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093d7594-abdc-4b9a-bb57-966e6e50129e" containerName="pull"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415882    4449 cpu_manager.go:414] "containerMap: removing container" podUID="50a250fd-1476-4dc2-b58c-87875aa93d6b" containerName="pull"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415889    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a250fd-1476-4dc2-b58c-87875aa93d6b" containerName="pull"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415901    4449 cpu_manager.go:414] "containerMap: removing container" podUID="093d7594-abdc-4b9a-bb57-966e6e50129e" containerName="util"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415921    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093d7594-abdc-4b9a-bb57-966e6e50129e" containerName="util"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415931    4449 cpu_manager.go:414] "containerMap: removing container" podUID="83bdaa43-45cb-48ae-911a-f9cece77b4ca" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.415939    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bdaa43-45cb-48ae-911a-f9cece77b4ca" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.416039    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9392dd-369a-4558-b4d9-e7b3852bbba3" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.416052    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="432e92b9-c705-4f37-9dbd-db584b1f5dfa" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.416064    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a250fd-1476-4dc2-b58c-87875aa93d6b" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.416072    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b4c77e-436d-457e-bf06-3584adb6794f" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.416081    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9efa6d2-8c8c-4505-b952-299b02c45f3e" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.416089    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="093d7594-abdc-4b9a-bb57-966e6e50129e" containerName="extract"
Nov 02 09:39:24.416118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.416101    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bdaa43-45cb-48ae-911a-f9cece77b4ca" containerName="extract"
Nov 02 09:39:24.419248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.419205    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.435539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.435507    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 09:39:24.435658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.435549    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-ptvwz\""
Nov 02 09:39:24.435658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.435560    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 09:39:24.435658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.435575    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 09:39:24.435658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.435643    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 09:39:24.435858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.435729    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 09:39:24.453165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.453131    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"]
Nov 02 09:39:24.501970 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.501931    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgwxk\" (UniqueName: \"kubernetes.io/projected/14ae7976-d81d-4847-97fe-ebe0cac032ec-kube-api-access-hgwxk\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.501970 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.501972    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ae7976-d81d-4847-97fe-ebe0cac032ec-tmp\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.502192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.501993    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14ae7976-d81d-4847-97fe-ebe0cac032ec-manifests\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.502192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.502050    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14ae7976-d81d-4847-97fe-ebe0cac032ec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.502192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.502107    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/14ae7976-d81d-4847-97fe-ebe0cac032ec-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.502192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.502150    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14ae7976-d81d-4847-97fe-ebe0cac032ec-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.603193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.603151    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgwxk\" (UniqueName: \"kubernetes.io/projected/14ae7976-d81d-4847-97fe-ebe0cac032ec-kube-api-access-hgwxk\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.603193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.603199    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ae7976-d81d-4847-97fe-ebe0cac032ec-tmp\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.603484 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.603248    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14ae7976-d81d-4847-97fe-ebe0cac032ec-manifests\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.603484 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.603270    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14ae7976-d81d-4847-97fe-ebe0cac032ec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.603484 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.603291    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/14ae7976-d81d-4847-97fe-ebe0cac032ec-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.603484 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.603435    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14ae7976-d81d-4847-97fe-ebe0cac032ec-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.603784 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.603751    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/14ae7976-d81d-4847-97fe-ebe0cac032ec-manifests\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.605963 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.605931    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14ae7976-d81d-4847-97fe-ebe0cac032ec-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.606236 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.606067    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/14ae7976-d81d-4847-97fe-ebe0cac032ec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.606236 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.606182    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ae7976-d81d-4847-97fe-ebe0cac032ec-tmp\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.606337 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.606321    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/14ae7976-d81d-4847-97fe-ebe0cac032ec-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.618047 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.618019    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgwxk\" (UniqueName: \"kubernetes.io/projected/14ae7976-d81d-4847-97fe-ebe0cac032ec-kube-api-access-hgwxk\") pod \"opendatahub-operator-controller-manager-f7698d755-kkqg6\" (UID: \"14ae7976-d81d-4847-97fe-ebe0cac032ec\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:24.732155 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:24.732063    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:25.029281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:25.029251    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"]
Nov 02 09:39:25.030782 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:39:25.030739    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ae7976_d81d_4847_97fe_ebe0cac032ec.slice/crio-9fbe03828984d400c00c2a3223e6e5b63f367b9e22a7189ac1884b9283f42471 WatchSource:0}: Error finding container 9fbe03828984d400c00c2a3223e6e5b63f367b9e22a7189ac1884b9283f42471: Status 404 returned error can't find the container with id 9fbe03828984d400c00c2a3223e6e5b63f367b9e22a7189ac1884b9283f42471
Nov 02 09:39:25.414261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:25.414204    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6" event={"ID":"14ae7976-d81d-4847-97fe-ebe0cac032ec","Type":"ContainerStarted","Data":"9fbe03828984d400c00c2a3223e6e5b63f367b9e22a7189ac1884b9283f42471"}
Nov 02 09:39:28.429770 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:28.429731    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6" event={"ID":"14ae7976-d81d-4847-97fe-ebe0cac032ec","Type":"ContainerStarted","Data":"50b46f67865008de268ee69e0aec649bc36f3c6c226bf4f3b8d110fa57742a08"}
Nov 02 09:39:29.435458 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:29.435413    4449 generic.go:356] "Generic (PLEG): container finished" podID="14ae7976-d81d-4847-97fe-ebe0cac032ec" containerID="50b46f67865008de268ee69e0aec649bc36f3c6c226bf4f3b8d110fa57742a08" exitCode=0
Nov 02 09:39:29.435924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:29.435453    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6" event={"ID":"14ae7976-d81d-4847-97fe-ebe0cac032ec","Type":"ContainerDied","Data":"50b46f67865008de268ee69e0aec649bc36f3c6c226bf4f3b8d110fa57742a08"}
Nov 02 09:39:30.441238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:30.441178    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6" event={"ID":"14ae7976-d81d-4847-97fe-ebe0cac032ec","Type":"ContainerStarted","Data":"a2c6f199d29488ba0c55e682d774240387a52d7d0dee4245e684866c01c84203"}
Nov 02 09:39:30.441651 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:30.441332    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:30.490585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:30.490523    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6" podStartSLOduration=2.206807096 podStartE2EDuration="6.490507391s" totalImagesPullingTime="3.806015843s" totalInitContainerRuntime="477.684452ms" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:30.487574476 +0000 UTC m=+247.425861264" watchObservedRunningTime="2026-11-02 09:39:30.490507391 +0000 UTC m=+247.428794166"
Nov 02 09:39:32.322904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:32.322862    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tjlwm"]
Nov 02 09:39:32.326365 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:32.326348    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tjlwm"
Nov 02 09:39:32.346127 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:32.346099    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tjlwm"]
Nov 02 09:39:32.347658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:32.347635    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 09:39:32.347960 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:32.347937    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 02 09:39:32.348034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:32.347974    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 09:39:32.348082 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:32.348041    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-b54kb\""
Nov 02 09:39:32.476192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:32.476151    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nktn\" (UniqueName: \"kubernetes.io/projected/11792e8e-2931-4725-aece-39791e1c90bb-kube-api-access-8nktn\") pod \"dns-operator-controller-manager-8557cbdb84-tjlwm\" (UID: \"11792e8e-2931-4725-aece-39791e1c90bb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tjlwm"
Nov 02 09:39:32.577499 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:32.577405    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nktn\" (UniqueName: \"kubernetes.io/projected/11792e8e-2931-4725-aece-39791e1c90bb-kube-api-access-8nktn\") pod \"dns-operator-controller-manager-8557cbdb84-tjlwm\" (UID: \"11792e8e-2931-4725-aece-39791e1c90bb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tjlwm"
Nov 02 09:39:32.595682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:32.595642    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nktn\" (UniqueName: \"kubernetes.io/projected/11792e8e-2931-4725-aece-39791e1c90bb-kube-api-access-8nktn\") pod \"dns-operator-controller-manager-8557cbdb84-tjlwm\" (UID: \"11792e8e-2931-4725-aece-39791e1c90bb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tjlwm"
Nov 02 09:39:32.638315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:32.638278    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tjlwm"
Nov 02 09:39:32.920470 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:32.920440    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tjlwm"]
Nov 02 09:39:32.922019 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:39:32.921991    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11792e8e_2931_4725_aece_39791e1c90bb.slice/crio-85061cc9618e0bd9de51452873cd40af092673e802f90c23b267134506e94905 WatchSource:0}: Error finding container 85061cc9618e0bd9de51452873cd40af092673e802f90c23b267134506e94905: Status 404 returned error can't find the container with id 85061cc9618e0bd9de51452873cd40af092673e802f90c23b267134506e94905
Nov 02 09:39:33.456638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:33.456600    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tjlwm" event={"ID":"11792e8e-2931-4725-aece-39791e1c90bb","Type":"ContainerStarted","Data":"85061cc9618e0bd9de51452873cd40af092673e802f90c23b267134506e94905"}
Nov 02 09:39:36.474963 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:36.474870    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tjlwm" event={"ID":"11792e8e-2931-4725-aece-39791e1c90bb","Type":"ContainerStarted","Data":"96a3ff5b5fccf05f4525108b34e7f7fd6b6dbd37bd8b7c6809ed133bd0d59436"}
Nov 02 09:39:36.475350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:36.474982    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tjlwm"
Nov 02 09:39:36.496379 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:36.496326    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tjlwm" podStartSLOduration=1.229827671 podStartE2EDuration="4.496311228s" totalImagesPullingTime="3.266483557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:36.493982341 +0000 UTC m=+253.432269140" watchObservedRunningTime="2026-11-02 09:39:36.496311228 +0000 UTC m=+253.434598003"
Nov 02 09:39:36.546203 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:39:36.546159    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:39:37.404645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:37.404604    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b66fk"]
Nov 02 09:39:37.408042 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:37.408023    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b66fk"
Nov 02 09:39:37.419947 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:37.419911    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-l7pg6\""
Nov 02 09:39:37.423289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:37.423261    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b66fk"]
Nov 02 09:39:37.522928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:37.522894    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqxrz\" (UniqueName: \"kubernetes.io/projected/36151d2a-760c-4b7c-bcee-daf2d6d3a562-kube-api-access-kqxrz\") pod \"authorino-operator-7b7bb4bc77-b66fk\" (UID: \"36151d2a-760c-4b7c-bcee-daf2d6d3a562\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b66fk"
Nov 02 09:39:37.623587 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:37.623552    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqxrz\" (UniqueName: \"kubernetes.io/projected/36151d2a-760c-4b7c-bcee-daf2d6d3a562-kube-api-access-kqxrz\") pod \"authorino-operator-7b7bb4bc77-b66fk\" (UID: \"36151d2a-760c-4b7c-bcee-daf2d6d3a562\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b66fk"
Nov 02 09:39:37.636643 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:37.636609    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqxrz\" (UniqueName: \"kubernetes.io/projected/36151d2a-760c-4b7c-bcee-daf2d6d3a562-kube-api-access-kqxrz\") pod \"authorino-operator-7b7bb4bc77-b66fk\" (UID: \"36151d2a-760c-4b7c-bcee-daf2d6d3a562\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b66fk"
Nov 02 09:39:37.720808 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:37.720715    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b66fk"
Nov 02 09:39:38.003249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:38.003198    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b66fk"]
Nov 02 09:39:38.005930 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:39:38.005889    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36151d2a_760c_4b7c_bcee_daf2d6d3a562.slice/crio-1e87adcbce4fc1bcac40423894906b31f91d0497b2996fd2946c897aa0cdbc77 WatchSource:0}: Error finding container 1e87adcbce4fc1bcac40423894906b31f91d0497b2996fd2946c897aa0cdbc77: Status 404 returned error can't find the container with id 1e87adcbce4fc1bcac40423894906b31f91d0497b2996fd2946c897aa0cdbc77
Nov 02 09:39:38.484607 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:38.484573    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b66fk" event={"ID":"36151d2a-760c-4b7c-bcee-daf2d6d3a562","Type":"ContainerStarted","Data":"1e87adcbce4fc1bcac40423894906b31f91d0497b2996fd2946c897aa0cdbc77"}
Nov 02 09:39:41.448317 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:41.448291    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-kkqg6"
Nov 02 09:39:43.284329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.284292    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp"]
Nov 02 09:39:43.287575 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.287555    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp"
Nov 02 09:39:43.291962 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.291937    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:39:43.292456 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.292435    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 02 09:39:43.292695 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.292678    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-crhcx\""
Nov 02 09:39:43.304338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.304316    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp"]
Nov 02 09:39:43.374329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.374277    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61ca8f33-b168-4f78-95d7-8927c750474b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp\" (UID: \"61ca8f33-b168-4f78-95d7-8927c750474b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp"
Nov 02 09:39:43.374525 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.374409    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t99db\" (UniqueName: \"kubernetes.io/projected/61ca8f33-b168-4f78-95d7-8927c750474b-kube-api-access-t99db\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp\" (UID: \"61ca8f33-b168-4f78-95d7-8927c750474b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp"
Nov 02 09:39:43.475528 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.475489    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61ca8f33-b168-4f78-95d7-8927c750474b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp\" (UID: \"61ca8f33-b168-4f78-95d7-8927c750474b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp"
Nov 02 09:39:43.475695 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.475552    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99db\" (UniqueName: \"kubernetes.io/projected/61ca8f33-b168-4f78-95d7-8927c750474b-kube-api-access-t99db\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp\" (UID: \"61ca8f33-b168-4f78-95d7-8927c750474b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp"
Nov 02 09:39:43.475961 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.475939    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61ca8f33-b168-4f78-95d7-8927c750474b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp\" (UID: \"61ca8f33-b168-4f78-95d7-8927c750474b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp"
Nov 02 09:39:43.506857 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.506819    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t99db\" (UniqueName: \"kubernetes.io/projected/61ca8f33-b168-4f78-95d7-8927c750474b-kube-api-access-t99db\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp\" (UID: \"61ca8f33-b168-4f78-95d7-8927c750474b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp"
Nov 02 09:39:43.599854 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.599821    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp"
Nov 02 09:39:43.881712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:43.881559    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp"]
Nov 02 09:39:43.884494 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:39:43.884458    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ca8f33_b168_4f78_95d7_8927c750474b.slice/crio-b49beee0191314b6bff662ebdc974760c9d97aa7313acfaae05c38a4e36034d2 WatchSource:0}: Error finding container b49beee0191314b6bff662ebdc974760c9d97aa7313acfaae05c38a4e36034d2: Status 404 returned error can't find the container with id b49beee0191314b6bff662ebdc974760c9d97aa7313acfaae05c38a4e36034d2
Nov 02 09:39:44.514994 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:44.514960    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp" event={"ID":"61ca8f33-b168-4f78-95d7-8927c750474b","Type":"ContainerStarted","Data":"b49beee0191314b6bff662ebdc974760c9d97aa7313acfaae05c38a4e36034d2"}
Nov 02 09:39:46.119314 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.119281    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"]
Nov 02 09:39:46.122672 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.122646    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"
Nov 02 09:39:46.131649 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.131607    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9mq74\""
Nov 02 09:39:46.138248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.138192    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"]
Nov 02 09:39:46.299110 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.299067    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-9vtdl\" (UID: \"bfd816be-80d6-49f5-b1a4-556e8a7e0f93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"
Nov 02 09:39:46.299346 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.299228    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9pm5\" (UniqueName: \"kubernetes.io/projected/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-kube-api-access-c9pm5\") pod \"kuadrant-operator-controller-manager-cf8469d7b-9vtdl\" (UID: \"bfd816be-80d6-49f5-b1a4-556e8a7e0f93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"
Nov 02 09:39:46.399986 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.399945    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9pm5\" (UniqueName: \"kubernetes.io/projected/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-kube-api-access-c9pm5\") pod \"kuadrant-operator-controller-manager-cf8469d7b-9vtdl\" (UID: \"bfd816be-80d6-49f5-b1a4-556e8a7e0f93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"
Nov 02 09:39:46.400160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.400004    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-9vtdl\" (UID: \"bfd816be-80d6-49f5-b1a4-556e8a7e0f93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"
Nov 02 09:39:46.400410 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.400391    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-9vtdl\" (UID: \"bfd816be-80d6-49f5-b1a4-556e8a7e0f93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"
Nov 02 09:39:46.424164 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.424131    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9pm5\" (UniqueName: \"kubernetes.io/projected/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-kube-api-access-c9pm5\") pod \"kuadrant-operator-controller-manager-cf8469d7b-9vtdl\" (UID: \"bfd816be-80d6-49f5-b1a4-556e8a7e0f93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"
Nov 02 09:39:46.436897 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.436854    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"
Nov 02 09:39:46.525895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.525853    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp" event={"ID":"61ca8f33-b168-4f78-95d7-8927c750474b","Type":"ContainerStarted","Data":"c1fcf1023436a02b8fda5e4a093941954d63e77ce348b741f8382c1ded337005"}
Nov 02 09:39:46.560594 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.560495    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n4vlp" podStartSLOduration=1.428968231 podStartE2EDuration="3.560480328s" totalImagesPullingTime="2.131512097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:46.558584892 +0000 UTC m=+263.496871685" watchObservedRunningTime="2026-11-02 09:39:46.560480328 +0000 UTC m=+263.498767104"
Nov 02 09:39:46.724022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:46.723994    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"]
Nov 02 09:39:46.726887 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:39:46.726855    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd816be_80d6_49f5_b1a4_556e8a7e0f93.slice/crio-dd99e2c3e8ca0188f91336d124b766a5012cf613afa017e703e02f4ae19d8030 WatchSource:0}: Error finding container dd99e2c3e8ca0188f91336d124b766a5012cf613afa017e703e02f4ae19d8030: Status 404 returned error can't find the container with id dd99e2c3e8ca0188f91336d124b766a5012cf613afa017e703e02f4ae19d8030
Nov 02 09:39:47.482243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:47.482179    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tjlwm"
Nov 02 09:39:47.532582 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:47.532539    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl" event={"ID":"bfd816be-80d6-49f5-b1a4-556e8a7e0f93","Type":"ContainerStarted","Data":"dd99e2c3e8ca0188f91336d124b766a5012cf613afa017e703e02f4ae19d8030"}
Nov 02 09:39:49.428941 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:49.428902    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2l8n5"]
Nov 02 09:39:49.466534 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:49.466493    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2l8n5"]
Nov 02 09:39:49.466723 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:49.466631    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2l8n5"
Nov 02 09:39:49.473892 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:49.473445    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-m7dfk\""
Nov 02 09:39:49.631984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:49.631948    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stgsv\" (UniqueName: \"kubernetes.io/projected/f719b895-f939-4354-b42a-9c252a5f7712-kube-api-access-stgsv\") pod \"limitador-operator-controller-manager-5778bb4449-2l8n5\" (UID: \"f719b895-f939-4354-b42a-9c252a5f7712\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2l8n5"
Nov 02 09:39:49.733344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:49.733251    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stgsv\" (UniqueName: \"kubernetes.io/projected/f719b895-f939-4354-b42a-9c252a5f7712-kube-api-access-stgsv\") pod \"limitador-operator-controller-manager-5778bb4449-2l8n5\" (UID: \"f719b895-f939-4354-b42a-9c252a5f7712\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2l8n5"
Nov 02 09:39:49.751024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:49.750986    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stgsv\" (UniqueName: \"kubernetes.io/projected/f719b895-f939-4354-b42a-9c252a5f7712-kube-api-access-stgsv\") pod \"limitador-operator-controller-manager-5778bb4449-2l8n5\" (UID: \"f719b895-f939-4354-b42a-9c252a5f7712\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2l8n5"
Nov 02 09:39:49.797597 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:49.797557    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2l8n5"
Nov 02 09:39:50.092707 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:50.092327    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-2l8n5"]
Nov 02 09:39:50.094376 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:39:50.094317    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf719b895_f939_4354_b42a_9c252a5f7712.slice/crio-fe81376b0bc79cdc3d0ca6fc20d1f50d94b2245e8e6e80c4a7d7ac8f1f1bbf3f WatchSource:0}: Error finding container fe81376b0bc79cdc3d0ca6fc20d1f50d94b2245e8e6e80c4a7d7ac8f1f1bbf3f: Status 404 returned error can't find the container with id fe81376b0bc79cdc3d0ca6fc20d1f50d94b2245e8e6e80c4a7d7ac8f1f1bbf3f
Nov 02 09:39:50.548415 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:50.548382    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2l8n5" event={"ID":"f719b895-f939-4354-b42a-9c252a5f7712","Type":"ContainerStarted","Data":"fe81376b0bc79cdc3d0ca6fc20d1f50d94b2245e8e6e80c4a7d7ac8f1f1bbf3f"}
Nov 02 09:39:51.556263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:51.556193    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b66fk" event={"ID":"36151d2a-760c-4b7c-bcee-daf2d6d3a562","Type":"ContainerStarted","Data":"14107ff08b42b24ffcf5ee6feeb965d40e63a68b1dbde7a45a5db54d4dfba825"}
Nov 02 09:39:51.556699 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:51.556389    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b66fk"
Nov 02 09:39:51.575504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:51.575457    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b66fk" podStartSLOduration=1.600460393 podStartE2EDuration="14.575439111s" totalImagesPullingTime="12.974978718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:51.574280004 +0000 UTC m=+268.512566798" watchObservedRunningTime="2026-11-02 09:39:51.575439111 +0000 UTC m=+268.513725884"
Nov 02 09:39:53.569464 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:53.569427    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2l8n5" event={"ID":"f719b895-f939-4354-b42a-9c252a5f7712","Type":"ContainerStarted","Data":"4f270c490584af8aebe6d349144bb01ac66ec376a04e322b300bc21a7fd0b7ab"}
Nov 02 09:39:53.569910 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:53.569590    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2l8n5"
Nov 02 09:39:53.601195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:53.601144    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2l8n5" podStartSLOduration=2.196229255 podStartE2EDuration="4.601130797s" totalImagesPullingTime="2.404901542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:53.600251911 +0000 UTC m=+270.538538690" watchObservedRunningTime="2026-11-02 09:39:53.601130797 +0000 UTC m=+270.539417573"
Nov 02 09:39:54.575828 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:54.575790    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl" event={"ID":"bfd816be-80d6-49f5-b1a4-556e8a7e0f93","Type":"ContainerStarted","Data":"bc6468d1353a126f3b70ea417e22b326b294bc779ab47852853b7c58d87d40bc"}
Nov 02 09:39:54.576279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:54.576037    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"
Nov 02 09:39:54.601196 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:54.601147    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl" podStartSLOduration=1.216737834 podStartE2EDuration="8.601135475s" totalImagesPullingTime="7.384397641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:54.599652611 +0000 UTC m=+271.537939406" watchObservedRunningTime="2026-11-02 09:39:54.601135475 +0000 UTC m=+271.539422250"
Nov 02 09:39:56.732973 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.732938    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"]
Nov 02 09:39:56.735662 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.735647    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.751020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.750983    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dcnsh\""
Nov 02 09:39:56.760106 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.760078    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"]
Nov 02 09:39:56.800904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.800862    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k2h2\" (UniqueName: \"kubernetes.io/projected/8edd99e2-b897-47c4-bdff-129cf994aad3-kube-api-access-6k2h2\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.801042 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.800914    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.801042 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.801016    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edd99e2-b897-47c4-bdff-129cf994aad3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.801118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.801064    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edd99e2-b897-47c4-bdff-129cf994aad3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.801155 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.801125    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.801155 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.801152    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.801245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.801174    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edd99e2-b897-47c4-bdff-129cf994aad3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.801245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.801193    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.801323 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.801277    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.902275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.902239    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edd99e2-b897-47c4-bdff-129cf994aad3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.902429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.902282    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.902429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.902298    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.902429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.902315    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edd99e2-b897-47c4-bdff-129cf994aad3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.902429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.902334    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.902429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.902371    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.902429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.902393    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k2h2\" (UniqueName: \"kubernetes.io/projected/8edd99e2-b897-47c4-bdff-129cf994aad3-kube-api-access-6k2h2\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.902429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.902427    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.902777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.902494    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edd99e2-b897-47c4-bdff-129cf994aad3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.902777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.902721    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.903028 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.902997    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.903144 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.903065    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8edd99e2-b897-47c4-bdff-129cf994aad3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.903144 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.903070    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.903335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.903311    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.905154 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.905132    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8edd99e2-b897-47c4-bdff-129cf994aad3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.906376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.906346    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8edd99e2-b897-47c4-bdff-129cf994aad3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.914810 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.914778    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8edd99e2-b897-47c4-bdff-129cf994aad3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:56.914896 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:56.914869    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k2h2\" (UniqueName: \"kubernetes.io/projected/8edd99e2-b897-47c4-bdff-129cf994aad3-kube-api-access-6k2h2\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585knkfm\" (UID: \"8edd99e2-b897-47c4-bdff-129cf994aad3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:57.047507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.047453    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:57.131958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.131927    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"]
Nov 02 09:39:57.135499 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.135478    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"
Nov 02 09:39:57.146949 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.146921    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 02 09:39:57.148521 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.148495    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"]
Nov 02 09:39:57.204795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.204757    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-52xrj\" (UID: \"7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"
Nov 02 09:39:57.204795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.204792    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-52xrj\" (UID: \"7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"
Nov 02 09:39:57.205034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.204898    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw5fj\" (UniqueName: \"kubernetes.io/projected/7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708-kube-api-access-xw5fj\") pod \"kuadrant-console-plugin-66c4fb575d-52xrj\" (UID: \"7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"
Nov 02 09:39:57.221741 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.221693    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zj9w7\""
Nov 02 09:39:57.221895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.221852    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 02 09:39:57.306084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.305997    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-52xrj\" (UID: \"7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"
Nov 02 09:39:57.306084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.306031    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-52xrj\" (UID: \"7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"
Nov 02 09:39:57.306084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.306054    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw5fj\" (UniqueName: \"kubernetes.io/projected/7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708-kube-api-access-xw5fj\") pod \"kuadrant-console-plugin-66c4fb575d-52xrj\" (UID: \"7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"
Nov 02 09:39:57.306746 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.306723    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-52xrj\" (UID: \"7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"
Nov 02 09:39:57.308896 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.308865    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-52xrj\" (UID: \"7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"
Nov 02 09:39:57.319975 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.319942    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw5fj\" (UniqueName: \"kubernetes.io/projected/7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708-kube-api-access-xw5fj\") pod \"kuadrant-console-plugin-66c4fb575d-52xrj\" (UID: \"7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"
Nov 02 09:39:57.349003 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.348980    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"]
Nov 02 09:39:57.352055 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:39:57.352019    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edd99e2_b897_47c4_bdff_129cf994aad3.slice/crio-568a74561e37468e0b899fdadee1e999dc8bd62b6ac14bf0a1d182e8e1c5e6bf WatchSource:0}: Error finding container 568a74561e37468e0b899fdadee1e999dc8bd62b6ac14bf0a1d182e8e1c5e6bf: Status 404 returned error can't find the container with id 568a74561e37468e0b899fdadee1e999dc8bd62b6ac14bf0a1d182e8e1c5e6bf
Nov 02 09:39:57.354635 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.354601    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:39:57.354738 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.354663    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:39:57.354738 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.354692    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:39:57.454197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.454155    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"
Nov 02 09:39:57.591951 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.591857    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm" event={"ID":"8edd99e2-b897-47c4-bdff-129cf994aad3","Type":"ContainerStarted","Data":"03e8e4e346cea4ae68d600d654f0b6ea1cc5da3668545efb21bdf3b3dd5ae298"}
Nov 02 09:39:57.592118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.591956    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm" event={"ID":"8edd99e2-b897-47c4-bdff-129cf994aad3","Type":"ContainerStarted","Data":"568a74561e37468e0b899fdadee1e999dc8bd62b6ac14bf0a1d182e8e1c5e6bf"}
Nov 02 09:39:57.733456 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.733380    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm" podStartSLOduration=1.733358499 podStartE2EDuration="1.733358499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:57.617707505 +0000 UTC m=+274.555994295" watchObservedRunningTime="2026-11-02 09:39:57.733358499 +0000 UTC m=+274.671645276"
Nov 02 09:39:57.735078 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:57.735056    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj"]
Nov 02 09:39:57.737164 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:39:57.737139    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7df6b4fa_47c6_4bd7_91fa_0dd2ad23a708.slice/crio-e5043747521f4f217b2d0d7ec5eeccacbc828ea8872d2e4d4e99eb3e578d5020 WatchSource:0}: Error finding container e5043747521f4f217b2d0d7ec5eeccacbc828ea8872d2e4d4e99eb3e578d5020: Status 404 returned error can't find the container with id e5043747521f4f217b2d0d7ec5eeccacbc828ea8872d2e4d4e99eb3e578d5020
Nov 02 09:39:58.047680 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:58.047645    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:58.052308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:58.052282    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:58.448991 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:58.448910    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dm2qp"]
Nov 02 09:39:58.451911 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:58.451887    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dm2qp"
Nov 02 09:39:58.468249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:58.468197    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tnpmb\""
Nov 02 09:39:58.470658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:58.470633    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dm2qp"]
Nov 02 09:39:58.517731 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:58.517701    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tf2j\" (UniqueName: \"kubernetes.io/projected/a6038f60-7ae3-462b-86e9-fdec22cbbf76-kube-api-access-7tf2j\") pod \"authorino-56c6c9ddd5-dm2qp\" (UID: \"a6038f60-7ae3-462b-86e9-fdec22cbbf76\") " pod="kuadrant-system/authorino-56c6c9ddd5-dm2qp"
Nov 02 09:39:58.597921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:58.597880    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj" event={"ID":"7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708","Type":"ContainerStarted","Data":"e5043747521f4f217b2d0d7ec5eeccacbc828ea8872d2e4d4e99eb3e578d5020"}
Nov 02 09:39:58.598269 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:58.598247    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:58.599362 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:58.599341    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585knkfm"
Nov 02 09:39:58.619316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:58.619276    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tf2j\" (UniqueName: \"kubernetes.io/projected/a6038f60-7ae3-462b-86e9-fdec22cbbf76-kube-api-access-7tf2j\") pod \"authorino-56c6c9ddd5-dm2qp\" (UID: \"a6038f60-7ae3-462b-86e9-fdec22cbbf76\") " pod="kuadrant-system/authorino-56c6c9ddd5-dm2qp"
Nov 02 09:39:58.634482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:58.634441    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tf2j\" (UniqueName: \"kubernetes.io/projected/a6038f60-7ae3-462b-86e9-fdec22cbbf76-kube-api-access-7tf2j\") pod \"authorino-56c6c9ddd5-dm2qp\" (UID: \"a6038f60-7ae3-462b-86e9-fdec22cbbf76\") " pod="kuadrant-system/authorino-56c6c9ddd5-dm2qp"
Nov 02 09:39:58.764348 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:58.764250    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dm2qp"
Nov 02 09:39:59.066383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:59.065391    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dm2qp"]
Nov 02 09:39:59.070738 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:39:59.070705    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6038f60_7ae3_462b_86e9_fdec22cbbf76.slice/crio-c785aea3cd7fd27a4e8c6023369aefa78cea13264faae963156a8775df416f32 WatchSource:0}: Error finding container c785aea3cd7fd27a4e8c6023369aefa78cea13264faae963156a8775df416f32: Status 404 returned error can't find the container with id c785aea3cd7fd27a4e8c6023369aefa78cea13264faae963156a8775df416f32
Nov 02 09:39:59.606171 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:59.606137    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dm2qp" event={"ID":"a6038f60-7ae3-462b-86e9-fdec22cbbf76","Type":"ContainerStarted","Data":"c785aea3cd7fd27a4e8c6023369aefa78cea13264faae963156a8775df416f32"}
Nov 02 09:39:59.995457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:59.995364    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"]
Nov 02 09:39:59.998300 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:39:59.998265    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.008700 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.008437    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 09:40:00.008700 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.008490    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 09:40:00.008700 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.008626    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-jtmb6\""
Nov 02 09:40:00.011288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.010750    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"]
Nov 02 09:40:00.011547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.011511    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 02 09:40:00.032312 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.032276    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cc56635-aa7a-4702-8575-2c73011a580f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-cwjgx\" (UID: \"1cc56635-aa7a-4702-8575-2c73011a580f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.032498 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.032411    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cc56635-aa7a-4702-8575-2c73011a580f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-cwjgx\" (UID: \"1cc56635-aa7a-4702-8575-2c73011a580f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.032498 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.032474    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1cc56635-aa7a-4702-8575-2c73011a580f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-cwjgx\" (UID: \"1cc56635-aa7a-4702-8575-2c73011a580f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.032614 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.032585    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7267\" (UniqueName: \"kubernetes.io/projected/1cc56635-aa7a-4702-8575-2c73011a580f-kube-api-access-n7267\") pod \"kserve-module-controller-manager-7566cf5fc8-cwjgx\" (UID: \"1cc56635-aa7a-4702-8575-2c73011a580f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.133400 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.133361    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cc56635-aa7a-4702-8575-2c73011a580f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-cwjgx\" (UID: \"1cc56635-aa7a-4702-8575-2c73011a580f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.133596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.133448    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cc56635-aa7a-4702-8575-2c73011a580f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-cwjgx\" (UID: \"1cc56635-aa7a-4702-8575-2c73011a580f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.133596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.133506    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1cc56635-aa7a-4702-8575-2c73011a580f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-cwjgx\" (UID: \"1cc56635-aa7a-4702-8575-2c73011a580f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.133596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.133536    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7267\" (UniqueName: \"kubernetes.io/projected/1cc56635-aa7a-4702-8575-2c73011a580f-kube-api-access-n7267\") pod \"kserve-module-controller-manager-7566cf5fc8-cwjgx\" (UID: \"1cc56635-aa7a-4702-8575-2c73011a580f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.133977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.133934    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cc56635-aa7a-4702-8575-2c73011a580f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-cwjgx\" (UID: \"1cc56635-aa7a-4702-8575-2c73011a580f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.133977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.133967    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1cc56635-aa7a-4702-8575-2c73011a580f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-cwjgx\" (UID: \"1cc56635-aa7a-4702-8575-2c73011a580f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.134420 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.134397    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cc56635-aa7a-4702-8575-2c73011a580f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-cwjgx\" (UID: \"1cc56635-aa7a-4702-8575-2c73011a580f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.150359 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.150316    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7267\" (UniqueName: \"kubernetes.io/projected/1cc56635-aa7a-4702-8575-2c73011a580f-kube-api-access-n7267\") pod \"kserve-module-controller-manager-7566cf5fc8-cwjgx\" (UID: \"1cc56635-aa7a-4702-8575-2c73011a580f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.315284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.315237    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:00.631363 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:00.631282    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"]
Nov 02 09:40:00.646162 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:40:00.646126    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc56635_aa7a_4702_8575_2c73011a580f.slice/crio-a4662859ea88cf224e753729d7f79f751a90b1464a7bd613f541b1f6a0aa6482 WatchSource:0}: Error finding container a4662859ea88cf224e753729d7f79f751a90b1464a7bd613f541b1f6a0aa6482: Status 404 returned error can't find the container with id a4662859ea88cf224e753729d7f79f751a90b1464a7bd613f541b1f6a0aa6482
Nov 02 09:40:01.622815 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:01.622768    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx" event={"ID":"1cc56635-aa7a-4702-8575-2c73011a580f","Type":"ContainerStarted","Data":"a4662859ea88cf224e753729d7f79f751a90b1464a7bd613f541b1f6a0aa6482"}
Nov 02 09:40:02.564161 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:02.564127    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b66fk"
Nov 02 09:40:02.633194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:02.633152    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dm2qp" event={"ID":"a6038f60-7ae3-462b-86e9-fdec22cbbf76","Type":"ContainerStarted","Data":"8020a91e7857455b6ddcfc2362bfd495fea3c5463f5e777ae91940e947567025"}
Nov 02 09:40:02.654510 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:02.654445    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-dm2qp" podStartSLOduration=2.040677027 podStartE2EDuration="4.654421307s" totalImagesPullingTime="2.61374428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:02.652264527 +0000 UTC m=+279.590551328" watchObservedRunningTime="2026-11-02 09:40:02.654421307 +0000 UTC m=+279.592708083"
Nov 02 09:40:03.640131 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:03.640097    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj" event={"ID":"7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708","Type":"ContainerStarted","Data":"a4a8e5504f8cc977103231605a9ffe429b969fc32e1d1900898f55bfcf2576c8"}
Nov 02 09:40:03.659599 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:03.659546    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-52xrj" podStartSLOduration=1.0650266 podStartE2EDuration="6.659530528s" totalImagesPullingTime="5.594503928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:03.658314746 +0000 UTC m=+280.596601522" watchObservedRunningTime="2026-11-02 09:40:03.659530528 +0000 UTC m=+280.597817303"
Nov 02 09:40:04.578113 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:04.578082    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-2l8n5"
Nov 02 09:40:04.647000 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:04.646966    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx" event={"ID":"1cc56635-aa7a-4702-8575-2c73011a580f","Type":"ContainerStarted","Data":"401ce1b9e923461f4fb768909ea3ba1d6128d793cc7e9ec2fa5e4e63dd10dc15"}
Nov 02 09:40:04.647435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:04.647191    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:04.667901 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:04.667848    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx" podStartSLOduration=2.688323025 podStartE2EDuration="5.667832548s" totalImagesPullingTime="2.979509523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:04.667096999 +0000 UTC m=+281.605383777" watchObservedRunningTime="2026-11-02 09:40:04.667832548 +0000 UTC m=+281.606119324"
Nov 02 09:40:05.583654 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:05.583619    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"
Nov 02 09:40:06.547459 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:40:06.547419    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:40:21.362926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.362891    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"]
Nov 02 09:40:21.370697 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.370669    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.378203 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.378174    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"]
Nov 02 09:40:21.380363 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.380331    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:40:21.381579 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.381558    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 09:40:21.381815 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.381799    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 09:40:21.381899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.381869    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-2dxmj\""
Nov 02 09:40:21.381943 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.381921    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 09:40:21.382076 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.382054    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 09:40:21.549691 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.549646    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00095481-c8a0-4520-984a-3f426e49a758-metrics-certs\") pod \"jobset-controller-manager-7cf664dd55-w9468\" (UID: \"00095481-c8a0-4520-984a-3f426e49a758\") " pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.549691 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.549700    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/00095481-c8a0-4520-984a-3f426e49a758-manager-config\") pod \"jobset-controller-manager-7cf664dd55-w9468\" (UID: \"00095481-c8a0-4520-984a-3f426e49a758\") " pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.549900 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.549806    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00095481-c8a0-4520-984a-3f426e49a758-cert\") pod \"jobset-controller-manager-7cf664dd55-w9468\" (UID: \"00095481-c8a0-4520-984a-3f426e49a758\") " pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.549900 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.549868    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzdn\" (UniqueName: \"kubernetes.io/projected/00095481-c8a0-4520-984a-3f426e49a758-kube-api-access-6kzdn\") pod \"jobset-controller-manager-7cf664dd55-w9468\" (UID: \"00095481-c8a0-4520-984a-3f426e49a758\") " pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.650882 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.650791    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzdn\" (UniqueName: \"kubernetes.io/projected/00095481-c8a0-4520-984a-3f426e49a758-kube-api-access-6kzdn\") pod \"jobset-controller-manager-7cf664dd55-w9468\" (UID: \"00095481-c8a0-4520-984a-3f426e49a758\") " pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.651033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.650889    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00095481-c8a0-4520-984a-3f426e49a758-metrics-certs\") pod \"jobset-controller-manager-7cf664dd55-w9468\" (UID: \"00095481-c8a0-4520-984a-3f426e49a758\") " pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.651033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.650921    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/00095481-c8a0-4520-984a-3f426e49a758-manager-config\") pod \"jobset-controller-manager-7cf664dd55-w9468\" (UID: \"00095481-c8a0-4520-984a-3f426e49a758\") " pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.651033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.650947    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00095481-c8a0-4520-984a-3f426e49a758-cert\") pod \"jobset-controller-manager-7cf664dd55-w9468\" (UID: \"00095481-c8a0-4520-984a-3f426e49a758\") " pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.651579 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.651556    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/00095481-c8a0-4520-984a-3f426e49a758-manager-config\") pod \"jobset-controller-manager-7cf664dd55-w9468\" (UID: \"00095481-c8a0-4520-984a-3f426e49a758\") " pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.653647 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.653628    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00095481-c8a0-4520-984a-3f426e49a758-cert\") pod \"jobset-controller-manager-7cf664dd55-w9468\" (UID: \"00095481-c8a0-4520-984a-3f426e49a758\") " pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.653707 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.653669    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00095481-c8a0-4520-984a-3f426e49a758-metrics-certs\") pod \"jobset-controller-manager-7cf664dd55-w9468\" (UID: \"00095481-c8a0-4520-984a-3f426e49a758\") " pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.665396 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.665368    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzdn\" (UniqueName: \"kubernetes.io/projected/00095481-c8a0-4520-984a-3f426e49a758-kube-api-access-6kzdn\") pod \"jobset-controller-manager-7cf664dd55-w9468\" (UID: \"00095481-c8a0-4520-984a-3f426e49a758\") " pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.684337 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.684311    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:21.968371 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:21.968346    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"]
Nov 02 09:40:21.970004 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:40:21.969976    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00095481_c8a0_4520_984a_3f426e49a758.slice/crio-664117f70227b78adc42ad759073fb6b2f39c60806f8a5c803cf4e09bdfe71fb WatchSource:0}: Error finding container 664117f70227b78adc42ad759073fb6b2f39c60806f8a5c803cf4e09bdfe71fb: Status 404 returned error can't find the container with id 664117f70227b78adc42ad759073fb6b2f39c60806f8a5c803cf4e09bdfe71fb
Nov 02 09:40:22.742080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:22.742045    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468" event={"ID":"00095481-c8a0-4520-984a-3f426e49a758","Type":"ContainerStarted","Data":"664117f70227b78adc42ad759073fb6b2f39c60806f8a5c803cf4e09bdfe71fb"}
Nov 02 09:40:23.622390 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:23.622361    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"]
Nov 02 09:40:23.622597 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:23.622556    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl" podUID="bfd816be-80d6-49f5-b1a4-556e8a7e0f93" containerName="manager" containerID="cri-o://bc6468d1353a126f3b70ea417e22b326b294bc779ab47852853b7c58d87d40bc" gracePeriod=10
Nov 02 09:40:23.749327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:23.749291    4449 generic.go:356] "Generic (PLEG): container finished" podID="bfd816be-80d6-49f5-b1a4-556e8a7e0f93" containerID="bc6468d1353a126f3b70ea417e22b326b294bc779ab47852853b7c58d87d40bc" exitCode=0
Nov 02 09:40:23.749696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:23.749363    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl" event={"ID":"bfd816be-80d6-49f5-b1a4-556e8a7e0f93","Type":"ContainerDied","Data":"bc6468d1353a126f3b70ea417e22b326b294bc779ab47852853b7c58d87d40bc"}
Nov 02 09:40:23.924230 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:23.924184    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"
Nov 02 09:40:24.072974 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:24.072941    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-extensions-socket-volume\") pod \"bfd816be-80d6-49f5-b1a4-556e8a7e0f93\" (UID: \"bfd816be-80d6-49f5-b1a4-556e8a7e0f93\") "
Nov 02 09:40:24.073145 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:24.073111    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-kube-api-access-c9pm5\" (UniqueName: \"kubernetes.io/projected/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-kube-api-access-c9pm5\") pod \"bfd816be-80d6-49f5-b1a4-556e8a7e0f93\" (UID: \"bfd816be-80d6-49f5-b1a4-556e8a7e0f93\") "
Nov 02 09:40:24.073458 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:24.073429    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-extensions-socket-volume" pod "bfd816be-80d6-49f5-b1a4-556e8a7e0f93" (UID: "bfd816be-80d6-49f5-b1a4-556e8a7e0f93"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:40:24.076098 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:24.076067    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-kube-api-access-c9pm5" pod "bfd816be-80d6-49f5-b1a4-556e8a7e0f93" (UID: "bfd816be-80d6-49f5-b1a4-556e8a7e0f93"). InnerVolumeSpecName "kube-api-access-c9pm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:40:24.174321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:24.174240    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9pm5\" (UniqueName: \"kubernetes.io/projected/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-kube-api-access-c9pm5\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:40:24.174321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:24.174277    4449 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfd816be-80d6-49f5-b1a4-556e8a7e0f93-extensions-socket-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:40:24.757532 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:24.757324    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl" event={"ID":"bfd816be-80d6-49f5-b1a4-556e8a7e0f93","Type":"ContainerDied","Data":"dd99e2c3e8ca0188f91336d124b766a5012cf613afa017e703e02f4ae19d8030"}
Nov 02 09:40:24.757532 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:24.757366    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"
Nov 02 09:40:24.757532 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:24.757409    4449 scope.go:122] "RemoveContainer" containerID="bc6468d1353a126f3b70ea417e22b326b294bc779ab47852853b7c58d87d40bc"
Nov 02 09:40:24.791475 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:24.791437    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"]
Nov 02 09:40:24.796636 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:24.796606    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-9vtdl"]
Nov 02 09:40:25.205958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:25.205921    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd816be-80d6-49f5-b1a4-556e8a7e0f93" path="/var/lib/kubelet/pods/bfd816be-80d6-49f5-b1a4-556e8a7e0f93/volumes"
Nov 02 09:40:25.653586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:25.653565    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cwjgx"
Nov 02 09:40:25.764695 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:25.764663    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468" event={"ID":"00095481-c8a0-4520-984a-3f426e49a758","Type":"ContainerStarted","Data":"dff96a0b47927c53aba969727f04e6895cfad6e235b9bfec55badc107562cef0"}
Nov 02 09:40:25.765007 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:25.764787    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:40:25.788469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:25.788422    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468" podStartSLOduration=1.107701653 podStartE2EDuration="4.788408171s" totalImagesPullingTime="3.680706518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:25.787525059 +0000 UTC m=+302.725811836" watchObservedRunningTime="2026-11-02 09:40:25.788408171 +0000 UTC m=+302.726694947"
Nov 02 09:40:36.545399 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:40:36.545316    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:40:36.775963 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:40:36.775933    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7cf664dd55-w9468"
Nov 02 09:41:06.546568 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:41:06.546523    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:41:36.546714 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:41:36.546663    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:41:56.232681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:56.232588    4449 generic.go:356] "Generic (PLEG): container finished" podID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerID="9b651472f6f70fe339a759e86aa53cbf0eafa4727ba38e18089fa52e7a1a6b0f" exitCode=0
Nov 02 09:41:56.232681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:56.232659    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" event={"ID":"e5c98f31-5358-46e5-825b-ab24b569f10b","Type":"ContainerDied","Data":"9b651472f6f70fe339a759e86aa53cbf0eafa4727ba38e18089fa52e7a1a6b0f"}
Nov 02 09:41:56.233122 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:56.233019    4449 scope.go:122] "RemoveContainer" containerID="9b651472f6f70fe339a759e86aa53cbf0eafa4727ba38e18089fa52e7a1a6b0f"
Nov 02 09:41:56.234127 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:56.234100    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:41:56.234195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:56.234166    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:41:56.234260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:56.234198    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:41:56.888205 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:56.888167    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-p8nf4"]
Nov 02 09:41:56.888760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:56.888742    4449 cpu_manager.go:414] "containerMap: removing container" podUID="bfd816be-80d6-49f5-b1a4-556e8a7e0f93" containerName="manager"
Nov 02 09:41:56.888806 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:56.888763    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd816be-80d6-49f5-b1a4-556e8a7e0f93" containerName="manager"
Nov 02 09:41:56.888896 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:56.888883    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd816be-80d6-49f5-b1a4-556e8a7e0f93" containerName="manager"
Nov 02 09:41:56.892182 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:56.892158    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-p8nf4"
Nov 02 09:41:56.911761 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:56.911728    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-p8nf4"]
Nov 02 09:41:56.979234 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:56.979165    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-p8nf4"]
Nov 02 09:41:56.979521 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:41:56.979491    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cq4md], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-p8nf4" podUID="66cf10cb-a9bb-4250-a2c8-920030914580"
Nov 02 09:41:57.006092 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.006059    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-tckkg"]
Nov 02 09:41:57.009413 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.009398    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-tckkg"
Nov 02 09:41:57.016069 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.016021    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 09:41:57.017542 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.017520    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-tckkg"]
Nov 02 09:41:57.021920 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.021890    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq4md\" (UniqueName: \"kubernetes.io/projected/66cf10cb-a9bb-4250-a2c8-920030914580-kube-api-access-cq4md\") pod \"authorino-6f6944bcdd-p8nf4\" (UID: \"66cf10cb-a9bb-4250-a2c8-920030914580\") " pod="kuadrant-system/authorino-6f6944bcdd-p8nf4"
Nov 02 09:41:57.123328 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.123299    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/49a84dd1-2339-49be-a25a-b789fb85548e-tls-cert\") pod \"authorino-6889cbd8c4-tckkg\" (UID: \"49a84dd1-2339-49be-a25a-b789fb85548e\") " pod="kuadrant-system/authorino-6889cbd8c4-tckkg"
Nov 02 09:41:57.123492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.123361    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq4md\" (UniqueName: \"kubernetes.io/projected/66cf10cb-a9bb-4250-a2c8-920030914580-kube-api-access-cq4md\") pod \"authorino-6f6944bcdd-p8nf4\" (UID: \"66cf10cb-a9bb-4250-a2c8-920030914580\") " pod="kuadrant-system/authorino-6f6944bcdd-p8nf4"
Nov 02 09:41:57.123492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.123440    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpq2k\" (UniqueName: \"kubernetes.io/projected/49a84dd1-2339-49be-a25a-b789fb85548e-kube-api-access-kpq2k\") pod \"authorino-6889cbd8c4-tckkg\" (UID: \"49a84dd1-2339-49be-a25a-b789fb85548e\") " pod="kuadrant-system/authorino-6889cbd8c4-tckkg"
Nov 02 09:41:57.138694 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.138616    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq4md\" (UniqueName: \"kubernetes.io/projected/66cf10cb-a9bb-4250-a2c8-920030914580-kube-api-access-cq4md\") pod \"authorino-6f6944bcdd-p8nf4\" (UID: \"66cf10cb-a9bb-4250-a2c8-920030914580\") " pod="kuadrant-system/authorino-6f6944bcdd-p8nf4"
Nov 02 09:41:57.224867 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.224835    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/49a84dd1-2339-49be-a25a-b789fb85548e-tls-cert\") pod \"authorino-6889cbd8c4-tckkg\" (UID: \"49a84dd1-2339-49be-a25a-b789fb85548e\") " pod="kuadrant-system/authorino-6889cbd8c4-tckkg"
Nov 02 09:41:57.225052 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.224921    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpq2k\" (UniqueName: \"kubernetes.io/projected/49a84dd1-2339-49be-a25a-b789fb85548e-kube-api-access-kpq2k\") pod \"authorino-6889cbd8c4-tckkg\" (UID: \"49a84dd1-2339-49be-a25a-b789fb85548e\") " pod="kuadrant-system/authorino-6889cbd8c4-tckkg"
Nov 02 09:41:57.227554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.227532    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/49a84dd1-2339-49be-a25a-b789fb85548e-tls-cert\") pod \"authorino-6889cbd8c4-tckkg\" (UID: \"49a84dd1-2339-49be-a25a-b789fb85548e\") " pod="kuadrant-system/authorino-6889cbd8c4-tckkg"
Nov 02 09:41:57.238446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.238409    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpq2k\" (UniqueName: \"kubernetes.io/projected/49a84dd1-2339-49be-a25a-b789fb85548e-kube-api-access-kpq2k\") pod \"authorino-6889cbd8c4-tckkg\" (UID: \"49a84dd1-2339-49be-a25a-b789fb85548e\") " pod="kuadrant-system/authorino-6889cbd8c4-tckkg"
Nov 02 09:41:57.240045 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.240026    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-p8nf4"
Nov 02 09:41:57.240045 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.240030    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" event={"ID":"e5c98f31-5358-46e5-825b-ab24b569f10b","Type":"ContainerStarted","Data":"5ef7cb2e4084b0be9cbca2bb936b5641639d23f375f9f23660d8ff6a34a42ad4"}
Nov 02 09:41:57.278278 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.278243    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-p8nf4"
Nov 02 09:41:57.295568 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.295541    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-tckkg"]
Nov 02 09:41:57.295769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.295757    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-tckkg"
Nov 02 09:41:57.331924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.331889    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57c4747b99-pjh9n"]
Nov 02 09:41:57.336181 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.335803    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c4747b99-pjh9n"
Nov 02 09:41:57.344640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.344376    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57c4747b99-pjh9n"]
Nov 02 09:41:57.427197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.427122    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66cf10cb-a9bb-4250-a2c8-920030914580-kube-api-access-cq4md\" (UniqueName: \"kubernetes.io/projected/66cf10cb-a9bb-4250-a2c8-920030914580-kube-api-access-cq4md\") pod \"66cf10cb-a9bb-4250-a2c8-920030914580\" (UID: \"66cf10cb-a9bb-4250-a2c8-920030914580\") "
Nov 02 09:41:57.427342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.427303    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa31be28-565f-40de-85df-3c79102c86d4-tls-cert\") pod \"authorino-57c4747b99-pjh9n\" (UID: \"fa31be28-565f-40de-85df-3c79102c86d4\") " pod="kuadrant-system/authorino-57c4747b99-pjh9n"
Nov 02 09:41:57.427504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.427483    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7488j\" (UniqueName: \"kubernetes.io/projected/fa31be28-565f-40de-85df-3c79102c86d4-kube-api-access-7488j\") pod \"authorino-57c4747b99-pjh9n\" (UID: \"fa31be28-565f-40de-85df-3c79102c86d4\") " pod="kuadrant-system/authorino-57c4747b99-pjh9n"
Nov 02 09:41:57.429493 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.429467    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cf10cb-a9bb-4250-a2c8-920030914580-kube-api-access-cq4md" pod "66cf10cb-a9bb-4250-a2c8-920030914580" (UID: "66cf10cb-a9bb-4250-a2c8-920030914580"). InnerVolumeSpecName "kube-api-access-cq4md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:41:57.528971 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.528936    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa31be28-565f-40de-85df-3c79102c86d4-tls-cert\") pod \"authorino-57c4747b99-pjh9n\" (UID: \"fa31be28-565f-40de-85df-3c79102c86d4\") " pod="kuadrant-system/authorino-57c4747b99-pjh9n"
Nov 02 09:41:57.529155 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.529037    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7488j\" (UniqueName: \"kubernetes.io/projected/fa31be28-565f-40de-85df-3c79102c86d4-kube-api-access-7488j\") pod \"authorino-57c4747b99-pjh9n\" (UID: \"fa31be28-565f-40de-85df-3c79102c86d4\") " pod="kuadrant-system/authorino-57c4747b99-pjh9n"
Nov 02 09:41:57.529155 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.529082    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq4md\" (UniqueName: \"kubernetes.io/projected/66cf10cb-a9bb-4250-a2c8-920030914580-kube-api-access-cq4md\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:41:57.531700 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.531676    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa31be28-565f-40de-85df-3c79102c86d4-tls-cert\") pod \"authorino-57c4747b99-pjh9n\" (UID: \"fa31be28-565f-40de-85df-3c79102c86d4\") " pod="kuadrant-system/authorino-57c4747b99-pjh9n"
Nov 02 09:41:57.541742 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.541710    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7488j\" (UniqueName: \"kubernetes.io/projected/fa31be28-565f-40de-85df-3c79102c86d4-kube-api-access-7488j\") pod \"authorino-57c4747b99-pjh9n\" (UID: \"fa31be28-565f-40de-85df-3c79102c86d4\") " pod="kuadrant-system/authorino-57c4747b99-pjh9n"
Nov 02 09:41:57.592146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.592113    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-tckkg"]
Nov 02 09:41:57.593501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.593472    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:41:57.593662 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.593641    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-24rkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Nov 02 09:41:57.593725 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.593693    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Nov 02 09:41:57.593789 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:41:57.593744    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49a84dd1_2339_49be_a25a_b789fb85548e.slice/crio-41319f518af991327edabd5b7c979095d54676095e8240386c4a934994cdcc86 WatchSource:0}: Error finding container 41319f518af991327edabd5b7c979095d54676095e8240386c4a934994cdcc86: Status 404 returned error can't find the container with id 41319f518af991327edabd5b7c979095d54676095e8240386c4a934994cdcc86
Nov 02 09:41:57.595232 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.595199    4449 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 09:41:57.670239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.665668    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c4747b99-pjh9n"
Nov 02 09:41:57.947468 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:57.947444    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57c4747b99-pjh9n"]
Nov 02 09:41:58.247262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:58.247160    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-tckkg" event={"ID":"49a84dd1-2339-49be-a25a-b789fb85548e","Type":"ContainerStarted","Data":"41319f518af991327edabd5b7c979095d54676095e8240386c4a934994cdcc86"}
Nov 02 09:41:58.248486 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:58.248457    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c4747b99-pjh9n" event={"ID":"fa31be28-565f-40de-85df-3c79102c86d4","Type":"ContainerStarted","Data":"c29b2a0ee2bcb2edbc77843dac1f3d2bbe60f6ecd14610dd510585391c2ce954"}
Nov 02 09:41:58.248632 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:58.248511    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-p8nf4"
Nov 02 09:41:58.306562 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:58.306477    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-p8nf4"]
Nov 02 09:41:58.308597 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:58.308572    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-p8nf4"]
Nov 02 09:41:58.594046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:58.594002    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-24rkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Nov 02 09:41:58.594265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:58.594068    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Nov 02 09:41:59.208395 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.208349    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66cf10cb-a9bb-4250-a2c8-920030914580" path="/var/lib/kubelet/pods/66cf10cb-a9bb-4250-a2c8-920030914580/volumes"
Nov 02 09:41:59.284949 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.284910    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-tckkg" event={"ID":"49a84dd1-2339-49be-a25a-b789fb85548e","Type":"ContainerStarted","Data":"2186bf951e68f59249b8803dbd9ffb1e03f83efde8c73aa8ae7af6debba06b66"}
Nov 02 09:41:59.285474 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.285058    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-tckkg" podUID="49a84dd1-2339-49be-a25a-b789fb85548e" containerName="authorino" containerID="cri-o://2186bf951e68f59249b8803dbd9ffb1e03f83efde8c73aa8ae7af6debba06b66" gracePeriod=30
Nov 02 09:41:59.287404 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.287372    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c4747b99-pjh9n" event={"ID":"fa31be28-565f-40de-85df-3c79102c86d4","Type":"ContainerStarted","Data":"655f6503a46a6a0004b29c9a93e8c462da2b123ff731636d00942785612ba05a"}
Nov 02 09:41:59.310233 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.305417    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-tckkg" podStartSLOduration=2.5082652149999998 podStartE2EDuration="3.305396141s" totalImagesPullingTime="797.130926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:41:59.302634099 +0000 UTC m=+396.240920902" watchObservedRunningTime="2026-11-02 09:41:59.305396141 +0000 UTC m=+396.243682918"
Nov 02 09:41:59.327981 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.327473    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57c4747b99-pjh9n" podStartSLOduration=1.6160688680000002 podStartE2EDuration="2.327451756s" totalImagesPullingTime="711.382888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:41:59.325637497 +0000 UTC m=+396.263924277" watchObservedRunningTime="2026-11-02 09:41:59.327451756 +0000 UTC m=+396.265738533"
Nov 02 09:41:59.380834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.380795    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dm2qp"]
Nov 02 09:41:59.381267 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.381232    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-dm2qp" podUID="a6038f60-7ae3-462b-86e9-fdec22cbbf76" containerName="authorino" containerID="cri-o://8020a91e7857455b6ddcfc2362bfd495fea3c5463f5e777ae91940e947567025" gracePeriod=30
Nov 02 09:41:59.596333 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.593657    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-24rkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Nov 02 09:41:59.596333 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.593714    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Nov 02 09:41:59.607007 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.606986    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-tckkg"
Nov 02 09:41:59.684306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.684275    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49a84dd1-2339-49be-a25a-b789fb85548e-tls-cert\" (UniqueName: \"kubernetes.io/secret/49a84dd1-2339-49be-a25a-b789fb85548e-tls-cert\") pod \"49a84dd1-2339-49be-a25a-b789fb85548e\" (UID: \"49a84dd1-2339-49be-a25a-b789fb85548e\") "
Nov 02 09:41:59.684444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.684336    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a84dd1-2339-49be-a25a-b789fb85548e-kube-api-access-kpq2k\" (UniqueName: \"kubernetes.io/projected/49a84dd1-2339-49be-a25a-b789fb85548e-kube-api-access-kpq2k\") pod \"49a84dd1-2339-49be-a25a-b789fb85548e\" (UID: \"49a84dd1-2339-49be-a25a-b789fb85548e\") "
Nov 02 09:41:59.686472 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.686446    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a84dd1-2339-49be-a25a-b789fb85548e-kube-api-access-kpq2k" pod "49a84dd1-2339-49be-a25a-b789fb85548e" (UID: "49a84dd1-2339-49be-a25a-b789fb85548e"). InnerVolumeSpecName "kube-api-access-kpq2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:41:59.687000 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.686983    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dm2qp"
Nov 02 09:41:59.695318 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.695295    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a84dd1-2339-49be-a25a-b789fb85548e-tls-cert" pod "49a84dd1-2339-49be-a25a-b789fb85548e" (UID: "49a84dd1-2339-49be-a25a-b789fb85548e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:41:59.785535 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.785510    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6038f60-7ae3-462b-86e9-fdec22cbbf76-kube-api-access-7tf2j\" (UniqueName: \"kubernetes.io/projected/a6038f60-7ae3-462b-86e9-fdec22cbbf76-kube-api-access-7tf2j\") pod \"a6038f60-7ae3-462b-86e9-fdec22cbbf76\" (UID: \"a6038f60-7ae3-462b-86e9-fdec22cbbf76\") "
Nov 02 09:41:59.785728 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.785715    4449 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/49a84dd1-2339-49be-a25a-b789fb85548e-tls-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:41:59.785803 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.785731    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpq2k\" (UniqueName: \"kubernetes.io/projected/49a84dd1-2339-49be-a25a-b789fb85548e-kube-api-access-kpq2k\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:41:59.787561 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.787532    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6038f60-7ae3-462b-86e9-fdec22cbbf76-kube-api-access-7tf2j" pod "a6038f60-7ae3-462b-86e9-fdec22cbbf76" (UID: "a6038f60-7ae3-462b-86e9-fdec22cbbf76"). InnerVolumeSpecName "kube-api-access-7tf2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:41:59.886669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:41:59.886614    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tf2j\" (UniqueName: \"kubernetes.io/projected/a6038f60-7ae3-462b-86e9-fdec22cbbf76-kube-api-access-7tf2j\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:42:00.300702 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.300662    4449 generic.go:356] "Generic (PLEG): container finished" podID="a6038f60-7ae3-462b-86e9-fdec22cbbf76" containerID="8020a91e7857455b6ddcfc2362bfd495fea3c5463f5e777ae91940e947567025" exitCode=0
Nov 02 09:42:00.301150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.300719    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-dm2qp"
Nov 02 09:42:00.301150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.300806    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dm2qp" event={"ID":"a6038f60-7ae3-462b-86e9-fdec22cbbf76","Type":"ContainerDied","Data":"8020a91e7857455b6ddcfc2362bfd495fea3c5463f5e777ae91940e947567025"}
Nov 02 09:42:00.301150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.300851    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-dm2qp" event={"ID":"a6038f60-7ae3-462b-86e9-fdec22cbbf76","Type":"ContainerDied","Data":"c785aea3cd7fd27a4e8c6023369aefa78cea13264faae963156a8775df416f32"}
Nov 02 09:42:00.301150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.300875    4449 scope.go:122] "RemoveContainer" containerID="8020a91e7857455b6ddcfc2362bfd495fea3c5463f5e777ae91940e947567025"
Nov 02 09:42:00.302311 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.302288    4449 generic.go:356] "Generic (PLEG): container finished" podID="49a84dd1-2339-49be-a25a-b789fb85548e" containerID="2186bf951e68f59249b8803dbd9ffb1e03f83efde8c73aa8ae7af6debba06b66" exitCode=0
Nov 02 09:42:00.302450 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.302402    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-tckkg" event={"ID":"49a84dd1-2339-49be-a25a-b789fb85548e","Type":"ContainerDied","Data":"2186bf951e68f59249b8803dbd9ffb1e03f83efde8c73aa8ae7af6debba06b66"}
Nov 02 09:42:00.302450 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.302440    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-tckkg" event={"ID":"49a84dd1-2339-49be-a25a-b789fb85548e","Type":"ContainerDied","Data":"41319f518af991327edabd5b7c979095d54676095e8240386c4a934994cdcc86"}
Nov 02 09:42:00.302538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.302409    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-tckkg"
Nov 02 09:42:00.311254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.311241    4449 scope.go:122] "RemoveContainer" containerID="8020a91e7857455b6ddcfc2362bfd495fea3c5463f5e777ae91940e947567025"
Nov 02 09:42:00.311614 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:42:00.311596    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8020a91e7857455b6ddcfc2362bfd495fea3c5463f5e777ae91940e947567025\": container with ID starting with 8020a91e7857455b6ddcfc2362bfd495fea3c5463f5e777ae91940e947567025 not found: ID does not exist" containerID="8020a91e7857455b6ddcfc2362bfd495fea3c5463f5e777ae91940e947567025"
Nov 02 09:42:00.311656 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.311623    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8020a91e7857455b6ddcfc2362bfd495fea3c5463f5e777ae91940e947567025"} err="failed to get container status \"8020a91e7857455b6ddcfc2362bfd495fea3c5463f5e777ae91940e947567025\": rpc error: code = NotFound desc = could not find container \"8020a91e7857455b6ddcfc2362bfd495fea3c5463f5e777ae91940e947567025\": container with ID starting with 8020a91e7857455b6ddcfc2362bfd495fea3c5463f5e777ae91940e947567025 not found: ID does not exist"
Nov 02 09:42:00.311656 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.311639    4449 scope.go:122] "RemoveContainer" containerID="2186bf951e68f59249b8803dbd9ffb1e03f83efde8c73aa8ae7af6debba06b66"
Nov 02 09:42:00.320000 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.319982    4449 scope.go:122] "RemoveContainer" containerID="2186bf951e68f59249b8803dbd9ffb1e03f83efde8c73aa8ae7af6debba06b66"
Nov 02 09:42:00.320273 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:42:00.320249    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2186bf951e68f59249b8803dbd9ffb1e03f83efde8c73aa8ae7af6debba06b66\": container with ID starting with 2186bf951e68f59249b8803dbd9ffb1e03f83efde8c73aa8ae7af6debba06b66 not found: ID does not exist" containerID="2186bf951e68f59249b8803dbd9ffb1e03f83efde8c73aa8ae7af6debba06b66"
Nov 02 09:42:00.320327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.320282    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2186bf951e68f59249b8803dbd9ffb1e03f83efde8c73aa8ae7af6debba06b66"} err="failed to get container status \"2186bf951e68f59249b8803dbd9ffb1e03f83efde8c73aa8ae7af6debba06b66\": rpc error: code = NotFound desc = could not find container \"2186bf951e68f59249b8803dbd9ffb1e03f83efde8c73aa8ae7af6debba06b66\": container with ID starting with 2186bf951e68f59249b8803dbd9ffb1e03f83efde8c73aa8ae7af6debba06b66 not found: ID does not exist"
Nov 02 09:42:00.324614 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.324593    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dm2qp"]
Nov 02 09:42:00.330788 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.330769    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-dm2qp"]
Nov 02 09:42:00.348783 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.348760    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-tckkg"]
Nov 02 09:42:00.377384 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.377349    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-tckkg"]
Nov 02 09:42:00.593795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.593759    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-24rkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Nov 02 09:42:00.593973 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:00.593819    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Nov 02 09:42:01.205628 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:01.205598    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a84dd1-2339-49be-a25a-b789fb85548e" path="/var/lib/kubelet/pods/49a84dd1-2339-49be-a25a-b789fb85548e/volumes"
Nov 02 09:42:01.205913 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:01.205900    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6038f60-7ae3-462b-86e9-fdec22cbbf76" path="/var/lib/kubelet/pods/a6038f60-7ae3-462b-86e9-fdec22cbbf76/volumes"
Nov 02 09:42:01.593528 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:01.593491    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-24rkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Nov 02 09:42:01.593887 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:01.593544    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Nov 02 09:42:02.597313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:02.597283    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:42:02.597708 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:02.597580    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:42:02.598205 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:02.598187    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 09:42:06.546023 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:42:06.545945    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:42:36.547475 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:42:36.547434    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:42:39.208136 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.208101    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"]
Nov 02 09:42:39.208785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.208636    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a6038f60-7ae3-462b-86e9-fdec22cbbf76" containerName="authorino"
Nov 02 09:42:39.208785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.208655    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6038f60-7ae3-462b-86e9-fdec22cbbf76" containerName="authorino"
Nov 02 09:42:39.208785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.208688    4449 cpu_manager.go:414] "containerMap: removing container" podUID="49a84dd1-2339-49be-a25a-b789fb85548e" containerName="authorino"
Nov 02 09:42:39.208785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.208696    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a84dd1-2339-49be-a25a-b789fb85548e" containerName="authorino"
Nov 02 09:42:39.208785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.208785    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a84dd1-2339-49be-a25a-b789fb85548e" containerName="authorino"
Nov 02 09:42:39.209029 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.208799    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6038f60-7ae3-462b-86e9-fdec22cbbf76" containerName="authorino"
Nov 02 09:42:39.231201 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.231164    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"]
Nov 02 09:42:39.231383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.231326    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"
Nov 02 09:42:39.309129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.309076    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ftlll\""
Nov 02 09:42:39.309770 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.309743    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 09:42:39.338368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.338331    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-j6sq9\" (UID: \"3c434dd2-b08d-4ecc-9ab2-d89ab694963b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"
Nov 02 09:42:39.338539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.338455    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn4wp\" (UniqueName: \"kubernetes.io/projected/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-kube-api-access-nn4wp\") pod \"payload-pre-processing-7b98bb678c-j6sq9\" (UID: \"3c434dd2-b08d-4ecc-9ab2-d89ab694963b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"
Nov 02 09:42:39.439903 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.439851    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-j6sq9\" (UID: \"3c434dd2-b08d-4ecc-9ab2-d89ab694963b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"
Nov 02 09:42:39.440109 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.439980    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn4wp\" (UniqueName: \"kubernetes.io/projected/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-kube-api-access-nn4wp\") pod \"payload-pre-processing-7b98bb678c-j6sq9\" (UID: \"3c434dd2-b08d-4ecc-9ab2-d89ab694963b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"
Nov 02 09:42:39.440755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.440732    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-j6sq9\" (UID: \"3c434dd2-b08d-4ecc-9ab2-d89ab694963b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"
Nov 02 09:42:39.452871 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.452826    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn4wp\" (UniqueName: \"kubernetes.io/projected/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-kube-api-access-nn4wp\") pod \"payload-pre-processing-7b98bb678c-j6sq9\" (UID: \"3c434dd2-b08d-4ecc-9ab2-d89ab694963b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"
Nov 02 09:42:39.566903 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.566861    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"
Nov 02 09:42:39.845538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:39.845508    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"]
Nov 02 09:42:39.848081 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:42:39.848045    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c434dd2_b08d_4ecc_9ab2_d89ab694963b.slice/crio-833b6a6d87f5fb42e79f719037e887f7ab210fe529ebea995f8b5963d9acc500 WatchSource:0}: Error finding container 833b6a6d87f5fb42e79f719037e887f7ab210fe529ebea995f8b5963d9acc500: Status 404 returned error can't find the container with id 833b6a6d87f5fb42e79f719037e887f7ab210fe529ebea995f8b5963d9acc500
Nov 02 09:42:40.510707 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:40.510666    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9" event={"ID":"3c434dd2-b08d-4ecc-9ab2-d89ab694963b","Type":"ContainerStarted","Data":"833b6a6d87f5fb42e79f719037e887f7ab210fe529ebea995f8b5963d9acc500"}
Nov 02 09:42:43.530019 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:43.529979    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9" event={"ID":"3c434dd2-b08d-4ecc-9ab2-d89ab694963b","Type":"ContainerStarted","Data":"616f43151913965db2fa4a00792856d1a3e2d98f19415d70f5d32bc4e39cde67"}
Nov 02 09:42:43.530459 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:43.530154    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"
Nov 02 09:42:43.550283 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:43.550210    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9" podStartSLOduration=1.575795096 podStartE2EDuration="4.550197041s" totalImagesPullingTime="2.974401945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:42:43.548784042 +0000 UTC m=+440.487070833" watchObservedRunningTime="2026-11-02 09:42:43.550197041 +0000 UTC m=+440.488483816"
Nov 02 09:42:54.537742 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:42:54.537711    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"
Nov 02 09:43:06.547941 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:43:06.547899    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:43:13.797403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:13.797365    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"]
Nov 02 09:43:13.805309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:13.805286    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:13.811324 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:13.811294    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 09:43:13.811324 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:13.811293    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 09:43:13.812966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:13.812085    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 02 09:43:13.812966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:13.812343    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cbg4l\""
Nov 02 09:43:13.813716 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:13.813695    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"]
Nov 02 09:43:13.963946 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:13.963906    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:13.964142 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:13.963957    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:13.964142 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:13.963980    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:13.964142 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:13.964097    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm9th\" (UniqueName: \"kubernetes.io/projected/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-kube-api-access-zm9th\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:13.964142 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:13.964118    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:13.964142 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:13.964138    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.064929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.064828    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.064929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.064888    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.064929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.064912    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.065210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.064973    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9th\" (UniqueName: \"kubernetes.io/projected/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-kube-api-access-zm9th\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.065210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.065000    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.065210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.065028    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.065414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.065344    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.065414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.065386    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.065527 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.065490    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.067399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.067367    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.067712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.067693    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.078591 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.078561    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm9th\" (UniqueName: \"kubernetes.io/projected/9c8f9db2-7c01-4c64-9424-bd7175d19fa2-kube-api-access-zm9th\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vzmpw\" (UID: \"9c8f9db2-7c01-4c64-9424-bd7175d19fa2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.121596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.121548    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:14.405326 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.405206    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"]
Nov 02 09:43:14.409203 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:43:14.409171    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8f9db2_7c01_4c64_9424_bd7175d19fa2.slice/crio-e5d4aa820322f3e15ef799a44fa7eb683bc8281ff3c499d03c67ca1909e35a77 WatchSource:0}: Error finding container e5d4aa820322f3e15ef799a44fa7eb683bc8281ff3c499d03c67ca1909e35a77: Status 404 returned error can't find the container with id e5d4aa820322f3e15ef799a44fa7eb683bc8281ff3c499d03c67ca1909e35a77
Nov 02 09:43:14.699275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:14.699157    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw" event={"ID":"9c8f9db2-7c01-4c64-9424-bd7175d19fa2","Type":"ContainerStarted","Data":"e5d4aa820322f3e15ef799a44fa7eb683bc8281ff3c499d03c67ca1909e35a77"}
Nov 02 09:43:15.883356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:15.883318    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"]
Nov 02 09:43:15.891975 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:15.891931    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:15.898424 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:15.898391    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"]
Nov 02 09:43:15.900937 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:15.900908    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 09:43:15.989564 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:15.989522    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:15.989710 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:15.989654    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:15.989710 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:15.989700    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:15.989802 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:15.989717    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:15.989802 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:15.989741    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htfwr\" (UniqueName: \"kubernetes.io/projected/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-kube-api-access-htfwr\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:15.989913 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:15.989901    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.090724 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.090674    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.090724 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.090727    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.090976 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.090759    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.090976 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.090791    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htfwr\" (UniqueName: \"kubernetes.io/projected/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-kube-api-access-htfwr\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.090976 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.090900    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.090976 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.090928    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.091421 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.091385    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.091546 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.091420    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.091546 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.091425    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.093512 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.093481    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.093811 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.093789    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.106554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.106465    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htfwr\" (UniqueName: \"kubernetes.io/projected/97fc91f5-ccd5-4f9d-8e45-592003d2ecd2-kube-api-access-htfwr\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6wwp5\" (UID: \"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.225962 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.225924    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:16.519037 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.519006    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"]
Nov 02 09:43:16.521353 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:43:16.521316    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fc91f5_ccd5_4f9d_8e45_592003d2ecd2.slice/crio-e8ca22900a16782fffa49bea0deb6a267ebab3c8d80fe4d98f43c2089b7c42e8 WatchSource:0}: Error finding container e8ca22900a16782fffa49bea0deb6a267ebab3c8d80fe4d98f43c2089b7c42e8: Status 404 returned error can't find the container with id e8ca22900a16782fffa49bea0deb6a267ebab3c8d80fe4d98f43c2089b7c42e8
Nov 02 09:43:16.714882 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.714834    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5" event={"ID":"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2","Type":"ContainerStarted","Data":"e8ca22900a16782fffa49bea0deb6a267ebab3c8d80fe4d98f43c2089b7c42e8"}
Nov 02 09:43:16.716876 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.716846    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw" event={"ID":"9c8f9db2-7c01-4c64-9424-bd7175d19fa2","Type":"ContainerStarted","Data":"dd0b72a3aa646cd4ffa1cecc2938bce13038d098ddd755e29212df00fd518e08"}
Nov 02 09:43:16.749334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:16.749266    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw" podStartSLOduration=2.312053667 podStartE2EDuration="3.749244654s" totalImagesPullingTime="1.437190987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:43:16.746814149 +0000 UTC m=+473.685100923" watchObservedRunningTime="2026-11-02 09:43:16.749244654 +0000 UTC m=+473.687531430"
Nov 02 09:43:17.122034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:17.121933    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:17.739334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:17.739280    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5" event={"ID":"97fc91f5-ccd5-4f9d-8e45-592003d2ecd2","Type":"ContainerStarted","Data":"e6827b23d440206a771a391c4e6a381e6bc8e97726d8e910ff8a9824e3721574"}
Nov 02 09:43:17.765525 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:17.765442    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5" podStartSLOduration=2.481070896 podStartE2EDuration="2.765415888s" totalImagesPullingTime="284.344992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:43:17.763803733 +0000 UTC m=+474.702090536" watchObservedRunningTime="2026-11-02 09:43:17.765415888 +0000 UTC m=+474.703702662"
Nov 02 09:43:18.226300 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:18.226255    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:19.115403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.115362    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-npjcg"]
Nov 02 09:43:19.119745 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.119717    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.134080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.133995    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 02 09:43:19.137545 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.137509    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-npjcg"]
Nov 02 09:43:19.233835 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.233789    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15cd868f-fb32-4dd9-bea2-e4e09323d130-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.234428 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.233936    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15cd868f-fb32-4dd9-bea2-e4e09323d130-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.234428 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.234021    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15cd868f-fb32-4dd9-bea2-e4e09323d130-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.234428 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.234158    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvlps\" (UniqueName: \"kubernetes.io/projected/15cd868f-fb32-4dd9-bea2-e4e09323d130-kube-api-access-rvlps\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.234428 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.234198    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15cd868f-fb32-4dd9-bea2-e4e09323d130-home\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.234428 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.234354    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15cd868f-fb32-4dd9-bea2-e4e09323d130-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.337795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.337741    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15cd868f-fb32-4dd9-bea2-e4e09323d130-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.338168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.338145    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15cd868f-fb32-4dd9-bea2-e4e09323d130-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.338355 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.338335    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15cd868f-fb32-4dd9-bea2-e4e09323d130-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.338489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.338473    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvlps\" (UniqueName: \"kubernetes.io/projected/15cd868f-fb32-4dd9-bea2-e4e09323d130-kube-api-access-rvlps\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.338595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.338582    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15cd868f-fb32-4dd9-bea2-e4e09323d130-home\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.338706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.338693    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15cd868f-fb32-4dd9-bea2-e4e09323d130-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.338864 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.338820    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15cd868f-fb32-4dd9-bea2-e4e09323d130-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.343240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.339886    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15cd868f-fb32-4dd9-bea2-e4e09323d130-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.343240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.340149    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15cd868f-fb32-4dd9-bea2-e4e09323d130-home\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.359411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.352440    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15cd868f-fb32-4dd9-bea2-e4e09323d130-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.359411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.353586    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15cd868f-fb32-4dd9-bea2-e4e09323d130-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.361095 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.361038    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvlps\" (UniqueName: \"kubernetes.io/projected/15cd868f-fb32-4dd9-bea2-e4e09323d130-kube-api-access-rvlps\") pod \"echo-simulator-kserve-64b6f8f96c-npjcg\" (UID: \"15cd868f-fb32-4dd9-bea2-e4e09323d130\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.442084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.441971    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:19.745115 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:19.745076    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-npjcg"]
Nov 02 09:43:19.746901 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:43:19.746864    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15cd868f_fb32_4dd9_bea2_e4e09323d130.slice/crio-db8f43909f140cf68e6623e23430439fc03b52ac8dcc50a4bf8e88b5fde59d3b WatchSource:0}: Error finding container db8f43909f140cf68e6623e23430439fc03b52ac8dcc50a4bf8e88b5fde59d3b: Status 404 returned error can't find the container with id db8f43909f140cf68e6623e23430439fc03b52ac8dcc50a4bf8e88b5fde59d3b
Nov 02 09:43:20.048628 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.048590    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"]
Nov 02 09:43:20.054094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.054056    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.070420 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.070283    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"]
Nov 02 09:43:20.084251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.080795    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 09:43:20.153489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.152952    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xc8b\" (UniqueName: \"kubernetes.io/projected/e47c04fc-e082-4ab7-8813-af36a87969b8-kube-api-access-2xc8b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.153489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.153005    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47c04fc-e082-4ab7-8813-af36a87969b8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.153489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.153039    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47c04fc-e082-4ab7-8813-af36a87969b8-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.153489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.153104    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47c04fc-e082-4ab7-8813-af36a87969b8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.153489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.153202    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47c04fc-e082-4ab7-8813-af36a87969b8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.153489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.153282    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47c04fc-e082-4ab7-8813-af36a87969b8-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.255416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.254146    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47c04fc-e082-4ab7-8813-af36a87969b8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.255416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.254246    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47c04fc-e082-4ab7-8813-af36a87969b8-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.255416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.254330    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xc8b\" (UniqueName: \"kubernetes.io/projected/e47c04fc-e082-4ab7-8813-af36a87969b8-kube-api-access-2xc8b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.255416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.254366    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47c04fc-e082-4ab7-8813-af36a87969b8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.255416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.254397    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47c04fc-e082-4ab7-8813-af36a87969b8-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.255416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.254464    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47c04fc-e082-4ab7-8813-af36a87969b8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.255416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.254936    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47c04fc-e082-4ab7-8813-af36a87969b8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.255416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.255162    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47c04fc-e082-4ab7-8813-af36a87969b8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.256319 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.255523    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47c04fc-e082-4ab7-8813-af36a87969b8-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.258936 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.258899    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47c04fc-e082-4ab7-8813-af36a87969b8-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.259191 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.259164    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47c04fc-e082-4ab7-8813-af36a87969b8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.276027 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.275983    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xc8b\" (UniqueName: \"kubernetes.io/projected/e47c04fc-e082-4ab7-8813-af36a87969b8-kube-api-access-2xc8b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w\" (UID: \"e47c04fc-e082-4ab7-8813-af36a87969b8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.392819 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.390580    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:20.699409 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.698492    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"]
Nov 02 09:43:20.711892 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:43:20.711841    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47c04fc_e082_4ab7_8813_af36a87969b8.slice/crio-9a4a010c0cb9a4cb69cf1113b1525e89fe34a3ef3acd853192f71dbd70ef2787 WatchSource:0}: Error finding container 9a4a010c0cb9a4cb69cf1113b1525e89fe34a3ef3acd853192f71dbd70ef2787: Status 404 returned error can't find the container with id 9a4a010c0cb9a4cb69cf1113b1525e89fe34a3ef3acd853192f71dbd70ef2787
Nov 02 09:43:20.763718 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.763666    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w" event={"ID":"e47c04fc-e082-4ab7-8813-af36a87969b8","Type":"ContainerStarted","Data":"9a4a010c0cb9a4cb69cf1113b1525e89fe34a3ef3acd853192f71dbd70ef2787"}
Nov 02 09:43:20.765710 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:20.765656    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg" event={"ID":"15cd868f-fb32-4dd9-bea2-e4e09323d130","Type":"ContainerStarted","Data":"db8f43909f140cf68e6623e23430439fc03b52ac8dcc50a4bf8e88b5fde59d3b"}
Nov 02 09:43:21.774055 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:21.773693    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w" event={"ID":"e47c04fc-e082-4ab7-8813-af36a87969b8","Type":"ContainerStarted","Data":"51ae4aa5b525d67c1d3bebac93740ffb82dd65daae64b48169866e8575dded90"}
Nov 02 09:43:21.794940 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:21.794885    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w" podStartSLOduration=1.473584984 podStartE2EDuration="1.794870674s" totalImagesPullingTime="321.28569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:43:21.794081399 +0000 UTC m=+478.732368202" watchObservedRunningTime="2026-11-02 09:43:21.794870674 +0000 UTC m=+478.733157450"
Nov 02 09:43:22.391695 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.391653    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:22.522646 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.522613    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"]
Nov 02 09:43:22.526982 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.526956    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.533381 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.533360    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 02 09:43:22.557117 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.554708    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"]
Nov 02 09:43:22.585107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.584721    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5b3557-3492-4152-9e0a-50291c7d1853-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.585107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.584786    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wthnr\" (UniqueName: \"kubernetes.io/projected/6e5b3557-3492-4152-9e0a-50291c7d1853-kube-api-access-wthnr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.585107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.584863    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5b3557-3492-4152-9e0a-50291c7d1853-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.585107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.584900    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5b3557-3492-4152-9e0a-50291c7d1853-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.585107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.584926    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5b3557-3492-4152-9e0a-50291c7d1853-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.585107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.584965    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5b3557-3492-4152-9e0a-50291c7d1853-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.687097 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.686503    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5b3557-3492-4152-9e0a-50291c7d1853-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.687446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.687424    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5b3557-3492-4152-9e0a-50291c7d1853-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.687986 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.687965    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5b3557-3492-4152-9e0a-50291c7d1853-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.688591 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.688571    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5b3557-3492-4152-9e0a-50291c7d1853-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.688758 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.688741    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthnr\" (UniqueName: \"kubernetes.io/projected/6e5b3557-3492-4152-9e0a-50291c7d1853-kube-api-access-wthnr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.688947 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.688932    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5b3557-3492-4152-9e0a-50291c7d1853-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.689379 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.689358    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5b3557-3492-4152-9e0a-50291c7d1853-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.690086 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.689748    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5b3557-3492-4152-9e0a-50291c7d1853-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.690197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.688473    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5b3557-3492-4152-9e0a-50291c7d1853-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.690916 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.690892    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5b3557-3492-4152-9e0a-50291c7d1853-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.695040 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.694992    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5b3557-3492-4152-9e0a-50291c7d1853-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.708748 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.708696    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wthnr\" (UniqueName: \"kubernetes.io/projected/6e5b3557-3492-4152-9e0a-50291c7d1853-kube-api-access-wthnr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w\" (UID: \"6e5b3557-3492-4152-9e0a-50291c7d1853\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:22.785458 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.785413    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg" event={"ID":"15cd868f-fb32-4dd9-bea2-e4e09323d130","Type":"ContainerStarted","Data":"b5cd05e8e4d185ee55d420640f7c178c675d97da3fd1c09b62b7ea3b53bbf0bc"}
Nov 02 09:43:22.839140 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:22.839107    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:23.136756 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:23.136711    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg" podStartSLOduration=1.961126554 podStartE2EDuration="4.136693733s" totalImagesPullingTime="2.175567179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:43:22.805828318 +0000 UTC m=+479.744115107" watchObservedRunningTime="2026-11-02 09:43:23.136693733 +0000 UTC m=+480.074980510"
Nov 02 09:43:23.137508 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:23.137489    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"]
Nov 02 09:43:23.138992 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:43:23.138967    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e5b3557_3492_4152_9e0a_50291c7d1853.slice/crio-e57aabaf673259dbd947910c8718e380ef4a996fcb55bd53d048b494fc36de31 WatchSource:0}: Error finding container e57aabaf673259dbd947910c8718e380ef4a996fcb55bd53d048b494fc36de31: Status 404 returned error can't find the container with id e57aabaf673259dbd947910c8718e380ef4a996fcb55bd53d048b494fc36de31
Nov 02 09:43:23.442911 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:23.442841    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:23.793587 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:23.793548    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" event={"ID":"6e5b3557-3492-4152-9e0a-50291c7d1853","Type":"ContainerStarted","Data":"85fb576cd3382dd61225ea95083df7e1dc1f52bd3c49f7b550fb16fa8a029c0b"}
Nov 02 09:43:23.794032 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:23.793604    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" event={"ID":"6e5b3557-3492-4152-9e0a-50291c7d1853","Type":"ContainerStarted","Data":"e57aabaf673259dbd947910c8718e380ef4a996fcb55bd53d048b494fc36de31"}
Nov 02 09:43:23.815289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:23.815245    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podStartSLOduration=1.498705349 podStartE2EDuration="1.815231076s" totalImagesPullingTime="316.525727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:43:23.813531886 +0000 UTC m=+480.751818693" watchObservedRunningTime="2026-11-02 09:43:23.815231076 +0000 UTC m=+480.753517842"
Nov 02 09:43:23.839429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:23.839390    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:24.121914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:24.121833    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:24.134889 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:24.134863    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:24.817192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:24.817155    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vzmpw"
Nov 02 09:43:26.226107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:26.226066    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:26.238804 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:26.238780    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:26.824316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:26.824287    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6wwp5"
Nov 02 09:43:29.442753 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:29.442718    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:29.455144 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:29.455123    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:29.841241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:29.841191    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-npjcg"
Nov 02 09:43:30.391575 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:30.391548    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:30.404952 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:30.404926    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:30.846642 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:30.846617    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w"
Nov 02 09:43:32.840088 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:32.840055    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:32.853165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:32.853139    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:32.864313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:32.864286    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:43:34.969896 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:34.969806    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-795bcb67b4-xgtc6"]
Nov 02 09:43:34.975471 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:34.975438    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-795bcb67b4-xgtc6"
Nov 02 09:43:34.982807 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:34.982779    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-795bcb67b4-xgtc6"]
Nov 02 09:43:35.117378 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:35.117331    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp89w\" (UniqueName: \"kubernetes.io/projected/f384c8c8-d8e1-447f-b53a-cf98f7103e28-kube-api-access-hp89w\") pod \"authorino-795bcb67b4-xgtc6\" (UID: \"f384c8c8-d8e1-447f-b53a-cf98f7103e28\") " pod="kuadrant-system/authorino-795bcb67b4-xgtc6"
Nov 02 09:43:35.117557 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:35.117467    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f384c8c8-d8e1-447f-b53a-cf98f7103e28-tls-cert\") pod \"authorino-795bcb67b4-xgtc6\" (UID: \"f384c8c8-d8e1-447f-b53a-cf98f7103e28\") " pod="kuadrant-system/authorino-795bcb67b4-xgtc6"
Nov 02 09:43:35.218171 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:35.218129    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f384c8c8-d8e1-447f-b53a-cf98f7103e28-tls-cert\") pod \"authorino-795bcb67b4-xgtc6\" (UID: \"f384c8c8-d8e1-447f-b53a-cf98f7103e28\") " pod="kuadrant-system/authorino-795bcb67b4-xgtc6"
Nov 02 09:43:35.218335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:35.218239    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp89w\" (UniqueName: \"kubernetes.io/projected/f384c8c8-d8e1-447f-b53a-cf98f7103e28-kube-api-access-hp89w\") pod \"authorino-795bcb67b4-xgtc6\" (UID: \"f384c8c8-d8e1-447f-b53a-cf98f7103e28\") " pod="kuadrant-system/authorino-795bcb67b4-xgtc6"
Nov 02 09:43:35.220878 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:35.220820    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f384c8c8-d8e1-447f-b53a-cf98f7103e28-tls-cert\") pod \"authorino-795bcb67b4-xgtc6\" (UID: \"f384c8c8-d8e1-447f-b53a-cf98f7103e28\") " pod="kuadrant-system/authorino-795bcb67b4-xgtc6"
Nov 02 09:43:35.232137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:35.232101    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp89w\" (UniqueName: \"kubernetes.io/projected/f384c8c8-d8e1-447f-b53a-cf98f7103e28-kube-api-access-hp89w\") pod \"authorino-795bcb67b4-xgtc6\" (UID: \"f384c8c8-d8e1-447f-b53a-cf98f7103e28\") " pod="kuadrant-system/authorino-795bcb67b4-xgtc6"
Nov 02 09:43:35.290861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:35.290819    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-795bcb67b4-xgtc6"
Nov 02 09:43:35.567305 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:35.567272    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-795bcb67b4-xgtc6"]
Nov 02 09:43:35.568407 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:43:35.568379    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf384c8c8_d8e1_447f_b53a_cf98f7103e28.slice/crio-e0d3856e218ec85009c7678e7e47b5cf82d7ea8fc135ee2e1eae6f01fcdc5879 WatchSource:0}: Error finding container e0d3856e218ec85009c7678e7e47b5cf82d7ea8fc135ee2e1eae6f01fcdc5879: Status 404 returned error can't find the container with id e0d3856e218ec85009c7678e7e47b5cf82d7ea8fc135ee2e1eae6f01fcdc5879
Nov 02 09:43:35.862670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:35.862577    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-795bcb67b4-xgtc6" event={"ID":"f384c8c8-d8e1-447f-b53a-cf98f7103e28","Type":"ContainerStarted","Data":"e0d3856e218ec85009c7678e7e47b5cf82d7ea8fc135ee2e1eae6f01fcdc5879"}
Nov 02 09:43:36.546969 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:43:36.546935    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:43:37.883786 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:37.883746    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-795bcb67b4-xgtc6" event={"ID":"f384c8c8-d8e1-447f-b53a-cf98f7103e28","Type":"ContainerStarted","Data":"cd1e750493d236b24c200bc8ad52013c4621bec7bd8b37885d75f593e2dcee11"}
Nov 02 09:43:37.906150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:37.906098    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-795bcb67b4-xgtc6" podStartSLOduration=2.658610363 podStartE2EDuration="3.906084601s" totalImagesPullingTime="1.247474238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:43:37.904929408 +0000 UTC m=+494.843216199" watchObservedRunningTime="2026-11-02 09:43:37.906084601 +0000 UTC m=+494.844371376"
Nov 02 09:43:37.940656 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:37.940618    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57c4747b99-pjh9n"]
Nov 02 09:43:37.941113 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:37.941079    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57c4747b99-pjh9n" podUID="fa31be28-565f-40de-85df-3c79102c86d4" containerName="authorino" containerID="cri-o://655f6503a46a6a0004b29c9a93e8c462da2b123ff731636d00942785612ba05a" gracePeriod=30
Nov 02 09:43:38.228710 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.228680    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c4747b99-pjh9n"
Nov 02 09:43:38.359153 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.359116    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa31be28-565f-40de-85df-3c79102c86d4-tls-cert\" (UniqueName: \"kubernetes.io/secret/fa31be28-565f-40de-85df-3c79102c86d4-tls-cert\") pod \"fa31be28-565f-40de-85df-3c79102c86d4\" (UID: \"fa31be28-565f-40de-85df-3c79102c86d4\") "
Nov 02 09:43:38.359334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.359272    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa31be28-565f-40de-85df-3c79102c86d4-kube-api-access-7488j\" (UniqueName: \"kubernetes.io/projected/fa31be28-565f-40de-85df-3c79102c86d4-kube-api-access-7488j\") pod \"fa31be28-565f-40de-85df-3c79102c86d4\" (UID: \"fa31be28-565f-40de-85df-3c79102c86d4\") "
Nov 02 09:43:38.361640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.361598    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa31be28-565f-40de-85df-3c79102c86d4-kube-api-access-7488j" pod "fa31be28-565f-40de-85df-3c79102c86d4" (UID: "fa31be28-565f-40de-85df-3c79102c86d4"). InnerVolumeSpecName "kube-api-access-7488j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:43:38.373254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.373192    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa31be28-565f-40de-85df-3c79102c86d4-tls-cert" pod "fa31be28-565f-40de-85df-3c79102c86d4" (UID: "fa31be28-565f-40de-85df-3c79102c86d4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:43:38.461195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.461095    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7488j\" (UniqueName: \"kubernetes.io/projected/fa31be28-565f-40de-85df-3c79102c86d4-kube-api-access-7488j\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:43:38.461195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.461137    4449 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa31be28-565f-40de-85df-3c79102c86d4-tls-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:43:38.890971 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.890938    4449 generic.go:356] "Generic (PLEG): container finished" podID="fa31be28-565f-40de-85df-3c79102c86d4" containerID="655f6503a46a6a0004b29c9a93e8c462da2b123ff731636d00942785612ba05a" exitCode=0
Nov 02 09:43:38.891477 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.890987    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c4747b99-pjh9n"
Nov 02 09:43:38.891477 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.891032    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c4747b99-pjh9n" event={"ID":"fa31be28-565f-40de-85df-3c79102c86d4","Type":"ContainerDied","Data":"655f6503a46a6a0004b29c9a93e8c462da2b123ff731636d00942785612ba05a"}
Nov 02 09:43:38.891477 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.891087    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c4747b99-pjh9n" event={"ID":"fa31be28-565f-40de-85df-3c79102c86d4","Type":"ContainerDied","Data":"c29b2a0ee2bcb2edbc77843dac1f3d2bbe60f6ecd14610dd510585391c2ce954"}
Nov 02 09:43:38.891477 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.891111    4449 scope.go:122] "RemoveContainer" containerID="655f6503a46a6a0004b29c9a93e8c462da2b123ff731636d00942785612ba05a"
Nov 02 09:43:38.904669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.904643    4449 scope.go:122] "RemoveContainer" containerID="655f6503a46a6a0004b29c9a93e8c462da2b123ff731636d00942785612ba05a"
Nov 02 09:43:38.905404 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:43:38.905365    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655f6503a46a6a0004b29c9a93e8c462da2b123ff731636d00942785612ba05a\": container with ID starting with 655f6503a46a6a0004b29c9a93e8c462da2b123ff731636d00942785612ba05a not found: ID does not exist" containerID="655f6503a46a6a0004b29c9a93e8c462da2b123ff731636d00942785612ba05a"
Nov 02 09:43:38.905527 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.905423    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655f6503a46a6a0004b29c9a93e8c462da2b123ff731636d00942785612ba05a"} err="failed to get container status \"655f6503a46a6a0004b29c9a93e8c462da2b123ff731636d00942785612ba05a\": rpc error: code = NotFound desc = could not find container \"655f6503a46a6a0004b29c9a93e8c462da2b123ff731636d00942785612ba05a\": container with ID starting with 655f6503a46a6a0004b29c9a93e8c462da2b123ff731636d00942785612ba05a not found: ID does not exist"
Nov 02 09:43:38.936948 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.936909    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57c4747b99-pjh9n"]
Nov 02 09:43:38.971497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:38.971467    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57c4747b99-pjh9n"]
Nov 02 09:43:39.206179 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:39.206086    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa31be28-565f-40de-85df-3c79102c86d4" path="/var/lib/kubelet/pods/fa31be28-565f-40de-85df-3c79102c86d4/volumes"
Nov 02 09:43:51.646801 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.646744    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"]
Nov 02 09:43:51.647300 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.647234    4449 cpu_manager.go:414] "containerMap: removing container" podUID="fa31be28-565f-40de-85df-3c79102c86d4" containerName="authorino"
Nov 02 09:43:51.647300 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.647251    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa31be28-565f-40de-85df-3c79102c86d4" containerName="authorino"
Nov 02 09:43:51.647375 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.647334    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa31be28-565f-40de-85df-3c79102c86d4" containerName="authorino"
Nov 02 09:43:51.654994 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.654963    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.668052 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.668006    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"]
Nov 02 09:43:51.705514 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.705466    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.705717 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.705538    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.705717 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.705630    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.705717 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.705664    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.705717 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.705686    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbrwk\" (UniqueName: \"kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-kube-api-access-bbrwk\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.705894 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.705747    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.705894 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.705841    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.705962 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.705929    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.705998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.705982    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.807020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.806969    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.807020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.807014    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.807442 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.807033    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbrwk\" (UniqueName: \"kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-kube-api-access-bbrwk\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.807442 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.807054    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.807442 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.807102    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.807442 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.807185    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.807442 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.807282    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.807442 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.807323    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.807442 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.807395    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.807781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.807694    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.807781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.807739    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.807883 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.807785    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.807947 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.807885    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.808044 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.808014    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.810034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.810002    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.810152 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.810086    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.821809 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.821780    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.821986 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.821957    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbrwk\" (UniqueName: \"kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-kube-api-access-bbrwk\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-f8wtz\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:51.972933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:51.972837    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:52.269264 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:52.269156    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"]
Nov 02 09:43:52.272885 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:43:52.272850    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60302c85_67c9_40da_b62c_1ff3ffdbcf1f.slice/crio-25e7beea7effd4df14500c2aeae7aa34cbdb9323ee2cb5651b28af3b4bf75877 WatchSource:0}: Error finding container 25e7beea7effd4df14500c2aeae7aa34cbdb9323ee2cb5651b28af3b4bf75877: Status 404 returned error can't find the container with id 25e7beea7effd4df14500c2aeae7aa34cbdb9323ee2cb5651b28af3b4bf75877
Nov 02 09:43:52.275087 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:52.275050    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:43:52.275184 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:52.275128    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:43:52.275184 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:52.275163    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:43:52.981235 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:52.981175    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz" event={"ID":"60302c85-67c9-40da-b62c-1ff3ffdbcf1f","Type":"ContainerStarted","Data":"ebc820c17c8afbe794d691337d245e870fcef4ff9c0ba9a18e6469b33aca93db"}
Nov 02 09:43:52.981641 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:52.981267    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz" event={"ID":"60302c85-67c9-40da-b62c-1ff3ffdbcf1f","Type":"ContainerStarted","Data":"25e7beea7effd4df14500c2aeae7aa34cbdb9323ee2cb5651b28af3b4bf75877"}
Nov 02 09:43:53.008387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:53.008330    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz" podStartSLOduration=2.008313617 podStartE2EDuration="2.008313617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:43:53.005667273 +0000 UTC m=+509.943954049" watchObservedRunningTime="2026-11-02 09:43:53.008313617 +0000 UTC m=+509.946600393"
Nov 02 09:43:53.974581 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:53.974519    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:43:53.976194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:53.976165    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 02 09:43:53.976353 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:53.976238    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz" podUID="60302c85-67c9-40da-b62c-1ff3ffdbcf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 02 09:43:54.973699 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:54.973652    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 02 09:43:54.974391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:54.974332    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz" podUID="60302c85-67c9-40da-b62c-1ff3ffdbcf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 02 09:43:55.973690 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:55.973652    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 02 09:43:55.974029 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:55.974001    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz" podUID="60302c85-67c9-40da-b62c-1ff3ffdbcf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 02 09:43:56.973690 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:56.973650    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 02 09:43:56.973994 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:56.973966    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz" podUID="60302c85-67c9-40da-b62c-1ff3ffdbcf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 02 09:43:57.974699 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:57.974661    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 02 09:43:57.975126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:57.974719    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz" podUID="60302c85-67c9-40da-b62c-1ff3ffdbcf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 02 09:43:58.973751 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:58.973711    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 02 09:43:58.973936 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:58.973772    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz" podUID="60302c85-67c9-40da-b62c-1ff3ffdbcf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 02 09:43:59.974119 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:59.974081    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 02 09:43:59.974537 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:43:59.974140    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz" podUID="60302c85-67c9-40da-b62c-1ff3ffdbcf1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 02 09:44:01.508892 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:01.508854    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:44:01.509381 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:01.509358    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:44:01.509997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:01.509978    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:44:06.549340 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:44:06.549297    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:44:06.687781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.687742    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"]
Nov 02 09:44:06.693366 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.693337    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.718333 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.718296    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"]
Nov 02 09:44:06.783285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.783233    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.783462 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.783295    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.783462 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.783339    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.783462 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.783367    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.783581 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.783469    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.783581 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.783563    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.783645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.783605    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.783645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.783633    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gktpd\" (UniqueName: \"kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-kube-api-access-gktpd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.783720 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.783702    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885010 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.884908    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885010 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.884956    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885010 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.884991    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885010 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.885012    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.885031    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.885053    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.885084    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.885111    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktpd\" (UniqueName: \"kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-kube-api-access-gktpd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.885143    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.885522    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.885632    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.885707    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885809 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.885760    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.885968 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.885950    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.888311 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.888288    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.888627 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.888602    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.899537 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.899500    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:06.899660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:06.899620    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktpd\" (UniqueName: \"kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-kube-api-access-gktpd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-h4446\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:07.008576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.008537    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:07.141709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.141620    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"]
Nov 02 09:44:07.145906 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.145876    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.155471 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.154957    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"]
Nov 02 09:44:07.156299 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.156262    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a110ef-openshift-default-dockercfg-qjmqj\""
Nov 02 09:44:07.261392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.261349    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"]
Nov 02 09:44:07.265098 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.265072    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.272908 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.272657    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2e80fb-openshift-default-dockercfg-545dt\""
Nov 02 09:44:07.291276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.291199    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.291455 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.291433    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"]
Nov 02 09:44:07.291592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.291575    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d365185-6209-4973-8a3f-52e66fe3181f-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.291629 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.291618    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29bf5\" (UniqueName: \"kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-kube-api-access-29bf5\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.291685 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.291650    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.291744 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.291685    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.291806 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.291792    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.291853 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.291832    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d365185-6209-4973-8a3f-52e66fe3181f-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.291942 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.291926    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.291997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.291961    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.316450 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.316414    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"]
Nov 02 09:44:07.317886 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:44:07.317858    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0a870c_883a_4314_9e4f_d05fb3241d5a.slice/crio-d66cc5c5af721fc8b5e094e172ab28e89dd862b974a436b26be61dc278cb7de0 WatchSource:0}: Error finding container d66cc5c5af721fc8b5e094e172ab28e89dd862b974a436b26be61dc278cb7de0: Status 404 returned error can't find the container with id d66cc5c5af721fc8b5e094e172ab28e89dd862b974a436b26be61dc278cb7de0
Nov 02 09:44:07.320096 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.320061    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:07.320172 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.320128    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:07.320172 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.320159    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:07.393368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393281    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d365185-6209-4973-8a3f-52e66fe3181f-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.393368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393321    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29bf5\" (UniqueName: \"kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-kube-api-access-29bf5\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.393368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393344    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.393660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393380    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.393660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393409    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.393660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393445    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.393660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393540    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.393660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393614    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.393906 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393683    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d365185-6209-4973-8a3f-52e66fe3181f-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.393906 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393728    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.393906 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393771    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.393906 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393805    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.393906 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393836    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.393906 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393878    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.394209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393918    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.394209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393925    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.394209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393945    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.394209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.393984    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.394209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.394174    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d365185-6209-4973-8a3f-52e66fe3181f-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.394495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.394235    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.394495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.394262    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwgzk\" (UniqueName: \"kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-kube-api-access-cwgzk\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.394495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.394286    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.394495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.394378    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.396532 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.396510    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.396699 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.396675    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d365185-6209-4973-8a3f-52e66fe3181f-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.410533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.410020    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29bf5\" (UniqueName: \"kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-kube-api-access-29bf5\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.411371 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.411339    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.465672 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.465379    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:07.496578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.495935    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.496578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.496021    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.496578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.496048    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwgzk\" (UniqueName: \"kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-kube-api-access-cwgzk\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.496578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.496079    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.496578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.496143    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.496578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.496172    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.496578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.496229    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.496578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.496298    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.496578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.496339    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.497842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.497120    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.497842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.497145    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.497842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.497438    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.497842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.497668    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.497842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.497813    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.499545 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.499521    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.500457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.500439    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.510462 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.510435    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwgzk\" (UniqueName: \"kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-kube-api-access-cwgzk\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.511287 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.511265    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.584149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.584105    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:07.771317 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.771282    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"]
Nov 02 09:44:07.771668 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:44:07.771646    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d365185_6209_4973_8a3f_52e66fe3181f.slice/crio-d67eddfccff727b8dc363c4963a7e009a279255de663eca91e73cdbdab5773e8 WatchSource:0}: Error finding container d67eddfccff727b8dc363c4963a7e009a279255de663eca91e73cdbdab5773e8: Status 404 returned error can't find the container with id d67eddfccff727b8dc363c4963a7e009a279255de663eca91e73cdbdab5773e8
Nov 02 09:44:07.773968 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.773931    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:07.774068 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.773997    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:07.774068 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.774026    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:07.898327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.897109    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"]
Nov 02 09:44:07.899833 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:44:07.899802    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfda7a4_ba6a_4941_9b13_a79d554fa88a.slice/crio-13ce58fbeb4ce2d6c1ef45a3579f0f41c2fa15531f34b13440a52e903330b18c WatchSource:0}: Error finding container 13ce58fbeb4ce2d6c1ef45a3579f0f41c2fa15531f34b13440a52e903330b18c: Status 404 returned error can't find the container with id 13ce58fbeb4ce2d6c1ef45a3579f0f41c2fa15531f34b13440a52e903330b18c
Nov 02 09:44:07.903296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.902250    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:07.903296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.902313    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:07.903296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:07.902382    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:08.104281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:08.090895    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" event={"ID":"8d365185-6209-4973-8a3f-52e66fe3181f","Type":"ContainerStarted","Data":"b5df30b3cc25eefccc19855a3ce5d91fb990e552d8ab9ee32bd0bb5ade864ac0"}
Nov 02 09:44:08.104281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:08.090985    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" event={"ID":"8d365185-6209-4973-8a3f-52e66fe3181f","Type":"ContainerStarted","Data":"d67eddfccff727b8dc363c4963a7e009a279255de663eca91e73cdbdab5773e8"}
Nov 02 09:44:08.111594 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:08.111486    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" event={"ID":"bcfda7a4-ba6a-4941-9b13-a79d554fa88a","Type":"ContainerStarted","Data":"eceb51ab9f2e960822f1c4ac856d6ee28714fa0e15601f108be40d068a7ba3cb"}
Nov 02 09:44:08.111594 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:08.111554    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" event={"ID":"bcfda7a4-ba6a-4941-9b13-a79d554fa88a","Type":"ContainerStarted","Data":"13ce58fbeb4ce2d6c1ef45a3579f0f41c2fa15531f34b13440a52e903330b18c"}
Nov 02 09:44:08.114471 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:08.113741    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" event={"ID":"dd0a870c-883a-4314-9e4f-d05fb3241d5a","Type":"ContainerStarted","Data":"f12027ca32095dbe15199fb3c209cfb642e4e2251ad89e9011172ea8b10ee92d"}
Nov 02 09:44:08.114471 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:08.113794    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" event={"ID":"dd0a870c-883a-4314-9e4f-d05fb3241d5a","Type":"ContainerStarted","Data":"d66cc5c5af721fc8b5e094e172ab28e89dd862b974a436b26be61dc278cb7de0"}
Nov 02 09:44:08.225296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:08.223712    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" podStartSLOduration=1.223691274 podStartE2EDuration="1.223691274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:08.154390688 +0000 UTC m=+525.092677466" watchObservedRunningTime="2026-11-02 09:44:08.223691274 +0000 UTC m=+525.161978050"
Nov 02 09:44:08.261731 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:08.261665    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" podStartSLOduration=2.261642311 podStartE2EDuration="2.261642311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:08.226154582 +0000 UTC m=+525.164441359" watchObservedRunningTime="2026-11-02 09:44:08.261642311 +0000 UTC m=+525.199929088"
Nov 02 09:44:08.266430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:08.266370    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" podStartSLOduration=1.266353692 podStartE2EDuration="1.266353692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:08.264571869 +0000 UTC m=+525.202858668" watchObservedRunningTime="2026-11-02 09:44:08.266353692 +0000 UTC m=+525.204640469"
Nov 02 09:44:08.473241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:08.469411    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:08.483410 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:08.483322    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:08.585072 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:08.585025    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:09.009090 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:09.009057    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:09.010349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:09.010313    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h4446 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 09:44:09.010471 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:09.010376    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" podUID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 09:44:09.119201 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:09.119172    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:09.120185 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:09.120165    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:09.590202 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:09.590173    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:10.009986 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:10.009910    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h4446 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 09:44:10.009986 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:10.009965    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" podUID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 09:44:10.125748 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:10.125709    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:10.126504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:10.126487    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:11.009394 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:11.009360    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h4446 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 09:44:11.009857 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:11.009421    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" podUID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 09:44:11.986553 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:11.986512    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"]
Nov 02 09:44:11.991577 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:11.991552    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"
Nov 02 09:44:12.007687 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.007661    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"]
Nov 02 09:44:12.009792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.009769    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h4446 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 09:44:12.010160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.009821    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" podUID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 09:44:12.029960 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.029921    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-a110ef-dockercfg-ncfwk\""
Nov 02 09:44:12.093889 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.093768    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-a110ef\""
Nov 02 09:44:12.094797 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.094752    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wr28\" (UniqueName: \"kubernetes.io/projected/6d32044d-b33f-4d9d-8e4a-c974bb485074-kube-api-access-5wr28\") pod \"payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh\" (UID: \"6d32044d-b33f-4d9d-8e4a-c974bb485074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"
Nov 02 09:44:12.095040 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.095023    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d32044d-b33f-4d9d-8e4a-c974bb485074-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh\" (UID: \"6d32044d-b33f-4d9d-8e4a-c974bb485074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"
Nov 02 09:44:12.196750 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.196704    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d32044d-b33f-4d9d-8e4a-c974bb485074-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh\" (UID: \"6d32044d-b33f-4d9d-8e4a-c974bb485074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"
Nov 02 09:44:12.196918 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.196901    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wr28\" (UniqueName: \"kubernetes.io/projected/6d32044d-b33f-4d9d-8e4a-c974bb485074-kube-api-access-5wr28\") pod \"payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh\" (UID: \"6d32044d-b33f-4d9d-8e4a-c974bb485074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"
Nov 02 09:44:12.197566 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.197537    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d32044d-b33f-4d9d-8e4a-c974bb485074-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh\" (UID: \"6d32044d-b33f-4d9d-8e4a-c974bb485074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"
Nov 02 09:44:12.214526 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.211094    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wr28\" (UniqueName: \"kubernetes.io/projected/6d32044d-b33f-4d9d-8e4a-c974bb485074-kube-api-access-5wr28\") pod \"payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh\" (UID: \"6d32044d-b33f-4d9d-8e4a-c974bb485074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"
Nov 02 09:44:12.311391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.311357    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"
Nov 02 09:44:12.625570 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.625543    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"]
Nov 02 09:44:12.626959 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:44:12.626934    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d32044d_b33f_4d9d_8e4a_c974bb485074.slice/crio-24c0624288d81219abdde050dc0696089c5d20faf65a1c0de13c82bcd467bda2 WatchSource:0}: Error finding container 24c0624288d81219abdde050dc0696089c5d20faf65a1c0de13c82bcd467bda2: Status 404 returned error can't find the container with id 24c0624288d81219abdde050dc0696089c5d20faf65a1c0de13c82bcd467bda2
Nov 02 09:44:12.995299 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.995182    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"]
Nov 02 09:44:12.999147 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:12.999123    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"
Nov 02 09:44:13.005141 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.005109    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2e80fb\""
Nov 02 09:44:13.006170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.006140    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec88150-3595-4245-a745-3e5cb6e0455d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j\" (UID: \"eec88150-3595-4245-a745-3e5cb6e0455d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"
Nov 02 09:44:13.006170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.006163    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"]
Nov 02 09:44:13.006367 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.006237    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dcwl\" (UniqueName: \"kubernetes.io/projected/eec88150-3595-4245-a745-3e5cb6e0455d-kube-api-access-4dcwl\") pod \"payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j\" (UID: \"eec88150-3595-4245-a745-3e5cb6e0455d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"
Nov 02 09:44:13.009489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.009465    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h4446 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 09:44:13.009596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.009501    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" podUID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 09:44:13.009820 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.009801    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"
Nov 02 09:44:13.018660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.018638    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"]
Nov 02 09:44:13.030284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.030261    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"]
Nov 02 09:44:13.107665 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.107611    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec88150-3595-4245-a745-3e5cb6e0455d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j\" (UID: \"eec88150-3595-4245-a745-3e5cb6e0455d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"
Nov 02 09:44:13.107841 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.107690    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dcwl\" (UniqueName: \"kubernetes.io/projected/eec88150-3595-4245-a745-3e5cb6e0455d-kube-api-access-4dcwl\") pod \"payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j\" (UID: \"eec88150-3595-4245-a745-3e5cb6e0455d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"
Nov 02 09:44:13.107841 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.107785    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9\" (UID: \"c2d7f4ec-14a0-450a-9298-c6da1c6fba89\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"
Nov 02 09:44:13.107841 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.107824    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79m6g\" (UniqueName: \"kubernetes.io/projected/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-kube-api-access-79m6g\") pod \"payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9\" (UID: \"c2d7f4ec-14a0-450a-9298-c6da1c6fba89\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"
Nov 02 09:44:13.108432 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.108412    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec88150-3595-4245-a745-3e5cb6e0455d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j\" (UID: \"eec88150-3595-4245-a745-3e5cb6e0455d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"
Nov 02 09:44:13.123917 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.123881    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dcwl\" (UniqueName: \"kubernetes.io/projected/eec88150-3595-4245-a745-3e5cb6e0455d-kube-api-access-4dcwl\") pod \"payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j\" (UID: \"eec88150-3595-4245-a745-3e5cb6e0455d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"
Nov 02 09:44:13.142904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.142859    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh" event={"ID":"6d32044d-b33f-4d9d-8e4a-c974bb485074","Type":"ContainerStarted","Data":"5df174620b4cd75f10c19d06a5db9d97cd3a5bcde6098dd82d111552c5f6ea56"}
Nov 02 09:44:13.143058 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.142933    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh" event={"ID":"6d32044d-b33f-4d9d-8e4a-c974bb485074","Type":"ContainerStarted","Data":"24c0624288d81219abdde050dc0696089c5d20faf65a1c0de13c82bcd467bda2"}
Nov 02 09:44:13.143058 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.142957    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"
Nov 02 09:44:13.176328 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.176255    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh" podStartSLOduration=2.176236308 podStartE2EDuration="2.176236308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:13.174447941 +0000 UTC m=+530.112734717" watchObservedRunningTime="2026-11-02 09:44:13.176236308 +0000 UTC m=+530.114523282"
Nov 02 09:44:13.209439 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.209396    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9\" (UID: \"c2d7f4ec-14a0-450a-9298-c6da1c6fba89\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"
Nov 02 09:44:13.209582 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.209456    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79m6g\" (UniqueName: \"kubernetes.io/projected/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-kube-api-access-79m6g\") pod \"payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9\" (UID: \"c2d7f4ec-14a0-450a-9298-c6da1c6fba89\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"
Nov 02 09:44:13.210113 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.210092    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9\" (UID: \"c2d7f4ec-14a0-450a-9298-c6da1c6fba89\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"
Nov 02 09:44:13.222100 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.222066    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79m6g\" (UniqueName: \"kubernetes.io/projected/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-kube-api-access-79m6g\") pod \"payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9\" (UID: \"c2d7f4ec-14a0-450a-9298-c6da1c6fba89\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"
Nov 02 09:44:13.312972 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.312934    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"
Nov 02 09:44:13.325439 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.325408    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"
Nov 02 09:44:13.659937 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.659903    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"]
Nov 02 09:44:13.660706 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:44:13.660664    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec88150_3595_4245_a745_3e5cb6e0455d.slice/crio-d5beda92820c6e65552fd304d65f4bfa225321a95b25fe5db060859355774e1c WatchSource:0}: Error finding container d5beda92820c6e65552fd304d65f4bfa225321a95b25fe5db060859355774e1c: Status 404 returned error can't find the container with id d5beda92820c6e65552fd304d65f4bfa225321a95b25fe5db060859355774e1c
Nov 02 09:44:13.661432 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:13.661408    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"]
Nov 02 09:44:13.664052 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:44:13.664029    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d7f4ec_14a0_450a_9298_c6da1c6fba89.slice/crio-af3696e0ced51607900d0706ce53629700d6543b7c620b4b042f99aeae4839e6 WatchSource:0}: Error finding container af3696e0ced51607900d0706ce53629700d6543b7c620b4b042f99aeae4839e6: Status 404 returned error can't find the container with id af3696e0ced51607900d0706ce53629700d6543b7c620b4b042f99aeae4839e6
Nov 02 09:44:14.009467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.009377    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h4446 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 09:44:14.009467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.009436    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" podUID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 09:44:14.159759 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.159714    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9" event={"ID":"c2d7f4ec-14a0-450a-9298-c6da1c6fba89","Type":"ContainerStarted","Data":"4067e0133864cc21ce45e35d3dd3338699b9d139773c83f2fd32969045d00a63"}
Nov 02 09:44:14.160205 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.159956    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"
Nov 02 09:44:14.160205 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.160001    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9" event={"ID":"c2d7f4ec-14a0-450a-9298-c6da1c6fba89","Type":"ContainerStarted","Data":"af3696e0ced51607900d0706ce53629700d6543b7c620b4b042f99aeae4839e6"}
Nov 02 09:44:14.161571 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.161541    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j" event={"ID":"eec88150-3595-4245-a745-3e5cb6e0455d","Type":"ContainerStarted","Data":"68196c16ea44175392cb383680a792ccef636a0653f024e93fb769a948314916"}
Nov 02 09:44:14.161709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.161595    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j" event={"ID":"eec88150-3595-4245-a745-3e5cb6e0455d","Type":"ContainerStarted","Data":"d5beda92820c6e65552fd304d65f4bfa225321a95b25fe5db060859355774e1c"}
Nov 02 09:44:14.190560 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.190499    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9" podStartSLOduration=2.19048512 podStartE2EDuration="2.19048512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:14.188781675 +0000 UTC m=+531.127068452" watchObservedRunningTime="2026-11-02 09:44:14.19048512 +0000 UTC m=+531.128771896"
Nov 02 09:44:14.217523 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.217462    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j" podStartSLOduration=2.217444567 podStartE2EDuration="2.217444567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:14.217163257 +0000 UTC m=+531.155450066" watchObservedRunningTime="2026-11-02 09:44:14.217444567 +0000 UTC m=+531.155731344"
Nov 02 09:44:14.416476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.416434    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"]
Nov 02 09:44:14.420392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.420366    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"
Nov 02 09:44:14.434864 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.434840    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"]
Nov 02 09:44:14.435062 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.435040    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-000ffb\""
Nov 02 09:44:14.435960 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.435324    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-000ffb-dockercfg-6fwjc\""
Nov 02 09:44:14.524606 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.524561    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whnw4\" (UniqueName: \"kubernetes.io/projected/2b246a3b-4f40-4174-b9a2-7f0669c1271a-kube-api-access-whnw4\") pod \"payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8\" (UID: \"2b246a3b-4f40-4174-b9a2-7f0669c1271a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"
Nov 02 09:44:14.524777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.524622    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b246a3b-4f40-4174-b9a2-7f0669c1271a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8\" (UID: \"2b246a3b-4f40-4174-b9a2-7f0669c1271a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"
Nov 02 09:44:14.626888 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.626845    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whnw4\" (UniqueName: \"kubernetes.io/projected/2b246a3b-4f40-4174-b9a2-7f0669c1271a-kube-api-access-whnw4\") pod \"payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8\" (UID: \"2b246a3b-4f40-4174-b9a2-7f0669c1271a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"
Nov 02 09:44:14.627058 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.626920    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b246a3b-4f40-4174-b9a2-7f0669c1271a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8\" (UID: \"2b246a3b-4f40-4174-b9a2-7f0669c1271a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"
Nov 02 09:44:14.632425 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.628003    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b246a3b-4f40-4174-b9a2-7f0669c1271a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8\" (UID: \"2b246a3b-4f40-4174-b9a2-7f0669c1271a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"
Nov 02 09:44:14.646973 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.646295    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whnw4\" (UniqueName: \"kubernetes.io/projected/2b246a3b-4f40-4174-b9a2-7f0669c1271a-kube-api-access-whnw4\") pod \"payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8\" (UID: \"2b246a3b-4f40-4174-b9a2-7f0669c1271a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"
Nov 02 09:44:14.738389 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:14.738288    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"
Nov 02 09:44:15.009343 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:15.009208    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h4446 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 09:44:15.009510 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:15.009352    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" podUID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 09:44:15.017262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:15.017231    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"]
Nov 02 09:44:15.018954 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:44:15.018918    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b246a3b_4f40_4174_b9a2_7f0669c1271a.slice/crio-d218a44e1cb74e1ba432831c5ae059999048c597c7ae3d1c586c5efb823099ce WatchSource:0}: Error finding container d218a44e1cb74e1ba432831c5ae059999048c597c7ae3d1c586c5efb823099ce: Status 404 returned error can't find the container with id d218a44e1cb74e1ba432831c5ae059999048c597c7ae3d1c586c5efb823099ce
Nov 02 09:44:15.169851 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:15.169819    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8" event={"ID":"2b246a3b-4f40-4174-b9a2-7f0669c1271a","Type":"ContainerStarted","Data":"17e65abb6ecb861ef50043b5b0a709be7288d408a690b9237dbc2e92898dfda6"}
Nov 02 09:44:15.170295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:15.169869    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8" event={"ID":"2b246a3b-4f40-4174-b9a2-7f0669c1271a","Type":"ContainerStarted","Data":"d218a44e1cb74e1ba432831c5ae059999048c597c7ae3d1c586c5efb823099ce"}
Nov 02 09:44:15.170295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:15.170242    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"
Nov 02 09:44:15.170295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:15.170271    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"
Nov 02 09:44:15.200547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:15.200495    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8" podStartSLOduration=1.200482297 podStartE2EDuration="1.200482297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:15.199694208 +0000 UTC m=+532.137980984" watchObservedRunningTime="2026-11-02 09:44:15.200482297 +0000 UTC m=+532.138769106"
Nov 02 09:44:16.956781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:16.956738    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:16.957511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:16.957469    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:16.958411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:16.958386    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:44:21.710060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.710029    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"]
Nov 02 09:44:21.714180 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.714150    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.729280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.729242    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"]
Nov 02 09:44:21.822448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.822379    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85060bf-426e-4a87-9e44-3f4577417345-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.822448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.822416    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.822448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.822437    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.822701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.822562    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.822701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.822590    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85060bf-426e-4a87-9e44-3f4577417345-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.822701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.822617    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.822701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.822694    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.822838 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.822743    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsg68\" (UniqueName: \"kubernetes.io/projected/e85060bf-426e-4a87-9e44-3f4577417345-kube-api-access-wsg68\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.822838 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.822766    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85060bf-426e-4a87-9e44-3f4577417345-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.923738 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.923702    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85060bf-426e-4a87-9e44-3f4577417345-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.923902 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.923752    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.923902 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.923778    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.923902 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.923834    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.923902 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.923857    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85060bf-426e-4a87-9e44-3f4577417345-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.923902 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.923892    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.924166 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.923926    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.924166 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.923985    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsg68\" (UniqueName: \"kubernetes.io/projected/e85060bf-426e-4a87-9e44-3f4577417345-kube-api-access-wsg68\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.924166 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.924020    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85060bf-426e-4a87-9e44-3f4577417345-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.924728 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.924701    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85060bf-426e-4a87-9e44-3f4577417345-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.925255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.925209    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.925538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.925516    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.925626 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.925596    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.925754 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.925734    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.927629 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.927606    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85060bf-426e-4a87-9e44-3f4577417345-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.927959 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.927940    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85060bf-426e-4a87-9e44-3f4577417345-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.937630 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.937593    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85060bf-426e-4a87-9e44-3f4577417345-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:21.937728 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:21.937702    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsg68\" (UniqueName: \"kubernetes.io/projected/e85060bf-426e-4a87-9e44-3f4577417345-kube-api-access-wsg68\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-qwf92\" (UID: \"e85060bf-426e-4a87-9e44-3f4577417345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:22.034998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:22.034967    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:22.348000 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:22.347972    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"]
Nov 02 09:44:22.348726 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:44:22.348687    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85060bf_426e_4a87_9e44_3f4577417345.slice/crio-e9ccb58b50a1dc4bcc60be1b59b603fb2971ffaff216a30d8bcc1564f33c47f3 WatchSource:0}: Error finding container e9ccb58b50a1dc4bcc60be1b59b603fb2971ffaff216a30d8bcc1564f33c47f3: Status 404 returned error can't find the container with id e9ccb58b50a1dc4bcc60be1b59b603fb2971ffaff216a30d8bcc1564f33c47f3
Nov 02 09:44:22.376836 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:22.376792    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:22.376958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:22.376886    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:22.376958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:22.376931    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:23.221183 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:23.221083    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" event={"ID":"e85060bf-426e-4a87-9e44-3f4577417345","Type":"ContainerStarted","Data":"c26a549b3777ffbcffc5ddbc05b029bff29723a9b5000f403f38ce6dc6d90641"}
Nov 02 09:44:23.221183 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:23.221162    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" event={"ID":"e85060bf-426e-4a87-9e44-3f4577417345","Type":"ContainerStarted","Data":"e9ccb58b50a1dc4bcc60be1b59b603fb2971ffaff216a30d8bcc1564f33c47f3"}
Nov 02 09:44:23.248624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:23.248563    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podStartSLOduration=2.248548373 podStartE2EDuration="2.248548373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:23.245925566 +0000 UTC m=+540.184212360" watchObservedRunningTime="2026-11-02 09:44:23.248548373 +0000 UTC m=+540.186835216"
Nov 02 09:44:24.035280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:24.035210    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:24.037120 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:24.037094    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 09:44:24.037278 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:24.037143    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 09:44:24.176878 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:24.176844    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"
Nov 02 09:44:25.036043 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:25.036000    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 09:44:25.036605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:25.036077    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 09:44:25.165242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:25.161418    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:25.165242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:25.161481    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:25.186742 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:25.186405    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"
Nov 02 09:44:26.036099 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:26.035606    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 09:44:26.036099 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:26.035669    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 09:44:26.205259 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:26.190568    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"
Nov 02 09:44:26.205259 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:26.200704    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"
Nov 02 09:44:27.036126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.036080    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 09:44:27.036708 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.036137    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 09:44:27.606409 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.606356    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"]
Nov 02 09:44:27.615474 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.615448    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"
Nov 02 09:44:27.628691 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.628659    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"]
Nov 02 09:44:27.630822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.630479    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:44:27.630822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.630517    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:44:27.630822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.630614    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:44:27.630822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.630680    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:44:27.736980 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.736930    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a32d6aa-67s9z\" (UID: \"180a67b5-4edb-4572-b71c-5913c10a34e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"
Nov 02 09:44:27.737188 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.737013    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a32d6aa-67s9z\" (UID: \"180a67b5-4edb-4572-b71c-5913c10a34e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"
Nov 02 09:44:27.737188 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.737140    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kt7r\" (UniqueName: \"kubernetes.io/projected/180a67b5-4edb-4572-b71c-5913c10a34e3-kube-api-access-4kt7r\") pod \"maas-api-revoke-sub-1a32d6aa-67s9z\" (UID: \"180a67b5-4edb-4572-b71c-5913c10a34e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"
Nov 02 09:44:27.838518 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.838465    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a32d6aa-67s9z\" (UID: \"180a67b5-4edb-4572-b71c-5913c10a34e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"
Nov 02 09:44:27.838794 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.838554    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a32d6aa-67s9z\" (UID: \"180a67b5-4edb-4572-b71c-5913c10a34e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"
Nov 02 09:44:27.838794 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.838674    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kt7r\" (UniqueName: \"kubernetes.io/projected/180a67b5-4edb-4572-b71c-5913c10a34e3-kube-api-access-4kt7r\") pod \"maas-api-revoke-sub-1a32d6aa-67s9z\" (UID: \"180a67b5-4edb-4572-b71c-5913c10a34e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"
Nov 02 09:44:27.839908 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.839849    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a32d6aa-67s9z\" (UID: \"180a67b5-4edb-4572-b71c-5913c10a34e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"
Nov 02 09:44:27.845356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.845323    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a32d6aa-67s9z\" (UID: \"180a67b5-4edb-4572-b71c-5913c10a34e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"
Nov 02 09:44:27.902161 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.902078    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kt7r\" (UniqueName: \"kubernetes.io/projected/180a67b5-4edb-4572-b71c-5913c10a34e3-kube-api-access-4kt7r\") pod \"maas-api-revoke-sub-1a32d6aa-67s9z\" (UID: \"180a67b5-4edb-4572-b71c-5913c10a34e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"
Nov 02 09:44:27.942321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:27.942273    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"
Nov 02 09:44:28.036071 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:28.036027    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 09:44:28.036396 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:28.036355    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 09:44:28.316705 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:28.316054    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"]
Nov 02 09:44:28.319014 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:44:28.318986    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180a67b5_4edb_4572_b71c_5913c10a34e3.slice/crio-6a6d9e10472125d7b36cba9ea19b89c07434c612576db289680590645ab297b2 WatchSource:0}: Error finding container 6a6d9e10472125d7b36cba9ea19b89c07434c612576db289680590645ab297b2: Status 404 returned error can't find the container with id 6a6d9e10472125d7b36cba9ea19b89c07434c612576db289680590645ab297b2
Nov 02 09:44:29.036094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:29.036054    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 09:44:29.036351 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:29.036119    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 09:44:29.310899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:29.310863    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z" event={"ID":"180a67b5-4edb-4572-b71c-5913c10a34e3","Type":"ContainerStarted","Data":"6a6d9e10472125d7b36cba9ea19b89c07434c612576db289680590645ab297b2"}
Nov 02 09:44:29.312914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:29.312884    4449 generic.go:356] "Generic (PLEG): container finished" podID="8d365185-6209-4973-8a3f-52e66fe3181f" containerID="b5df30b3cc25eefccc19855a3ce5d91fb990e552d8ab9ee32bd0bb5ade864ac0" exitCode=0
Nov 02 09:44:29.313036 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:29.312952    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" event={"ID":"8d365185-6209-4973-8a3f-52e66fe3181f","Type":"ContainerDied","Data":"b5df30b3cc25eefccc19855a3ce5d91fb990e552d8ab9ee32bd0bb5ade864ac0"}
Nov 02 09:44:29.313516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:29.313494    4449 scope.go:122] "RemoveContainer" containerID="b5df30b3cc25eefccc19855a3ce5d91fb990e552d8ab9ee32bd0bb5ade864ac0"
Nov 02 09:44:29.314984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:29.314955    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:29.315082 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:29.315028    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:29.315082 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:29.315062    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:30.035680 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:30.035632    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 09:44:30.035895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:30.035696    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 09:44:30.334583 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:30.334489    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" event={"ID":"8d365185-6209-4973-8a3f-52e66fe3181f","Type":"ContainerStarted","Data":"770df8433dab0e28bd44ad297b91bb41c46174a96da4c9e6cae3dbed5519542d"}
Nov 02 09:44:30.470280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:30.469389    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:30.470280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:30.470053    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 09:44:30.470280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:30.470132    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 09:44:31.036791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:31.035495    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 09:44:31.036791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:31.035559    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 09:44:31.467799 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:31.467759    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 09:44:31.468504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:31.468454    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 09:44:32.484247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:32.483381    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 09:44:32.484247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:32.483441    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 09:44:33.022243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:33.018325    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-h4446 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:33.022243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:33.018412    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" podUID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:33.094241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:33.093330    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:33.094241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:33.093396    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:33.469244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:33.468624    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 09:44:33.469244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:33.468683    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 09:44:33.657242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:33.642568    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-24rkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:33.657242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:33.642680    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:34.162243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:34.149384    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:34.162243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:34.149454    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:34.470296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:34.468245    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 09:44:34.470296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:34.468314    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 09:44:35.156348 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:35.155523    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:35.156348 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:35.155647    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:35.476305 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:35.474915    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 09:44:35.476305 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:35.474977    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 09:44:36.161243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:36.160324    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:36.161243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:36.160397    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:36.470294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:36.469676    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 09:44:36.470294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:36.469747    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 09:44:36.610405 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:44:36.610343    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:44:37.036507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:37.035884    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:37.205175 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:37.204439    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:37.205175 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:37.204518    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:37.469179 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:37.468662    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 09:44:37.469179 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:37.468728    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 09:44:38.248243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:38.247382    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:38.248243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:38.247472    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:38.579243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:38.578320    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:38.585253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:38.579733    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:38.585253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:38.582885    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 09:44:38.909295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:38.895399    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"]
Nov 02 09:44:38.936249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:38.916568    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"
Nov 02 09:44:39.003243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:38.968316    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d2e277\""
Nov 02 09:44:39.038595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.038386    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"]
Nov 02 09:44:39.108669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.108442    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72acb2-73e1-4e88-8639-de0fb5ddce30-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n\" (UID: \"7f72acb2-73e1-4e88-8639-de0fb5ddce30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"
Nov 02 09:44:39.108669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.108518    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4n6c\" (UniqueName: \"kubernetes.io/projected/7f72acb2-73e1-4e88-8639-de0fb5ddce30-kube-api-access-s4n6c\") pod \"payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n\" (UID: \"7f72acb2-73e1-4e88-8639-de0fb5ddce30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"
Nov 02 09:44:39.174905 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.174793    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"]
Nov 02 09:44:39.180318 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.180288    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"]
Nov 02 09:44:39.180787 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.180604    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"
Nov 02 09:44:39.185452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.185431    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d2e277-dockercfg-q2jhs\""
Nov 02 09:44:39.230669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.227860    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72acb2-73e1-4e88-8639-de0fb5ddce30-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n\" (UID: \"7f72acb2-73e1-4e88-8639-de0fb5ddce30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"
Nov 02 09:44:39.230669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.227943    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4n6c\" (UniqueName: \"kubernetes.io/projected/7f72acb2-73e1-4e88-8639-de0fb5ddce30-kube-api-access-s4n6c\") pod \"payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n\" (UID: \"7f72acb2-73e1-4e88-8639-de0fb5ddce30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"
Nov 02 09:44:39.230669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.228004    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ec8375-f77f-4b8f-a21c-9303968468ca-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv\" (UID: \"09ec8375-f77f-4b8f-a21c-9303968468ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"
Nov 02 09:44:39.230669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.228058    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvb5z\" (UniqueName: \"kubernetes.io/projected/09ec8375-f77f-4b8f-a21c-9303968468ca-kube-api-access-bvb5z\") pod \"payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv\" (UID: \"09ec8375-f77f-4b8f-a21c-9303968468ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"
Nov 02 09:44:39.230669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.229186    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72acb2-73e1-4e88-8639-de0fb5ddce30-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n\" (UID: \"7f72acb2-73e1-4e88-8639-de0fb5ddce30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"
Nov 02 09:44:39.251981 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.251849    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:39.252627 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.252592    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:39.349505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.344710    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvb5z\" (UniqueName: \"kubernetes.io/projected/09ec8375-f77f-4b8f-a21c-9303968468ca-kube-api-access-bvb5z\") pod \"payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv\" (UID: \"09ec8375-f77f-4b8f-a21c-9303968468ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"
Nov 02 09:44:39.349505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.344986    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ec8375-f77f-4b8f-a21c-9303968468ca-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv\" (UID: \"09ec8375-f77f-4b8f-a21c-9303968468ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"
Nov 02 09:44:39.349505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.345972    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ec8375-f77f-4b8f-a21c-9303968468ca-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv\" (UID: \"09ec8375-f77f-4b8f-a21c-9303968468ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"
Nov 02 09:44:39.350922 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.350856    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4n6c\" (UniqueName: \"kubernetes.io/projected/7f72acb2-73e1-4e88-8639-de0fb5ddce30-kube-api-access-s4n6c\") pod \"payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n\" (UID: \"7f72acb2-73e1-4e88-8639-de0fb5ddce30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"
Nov 02 09:44:39.386244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.386164    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvb5z\" (UniqueName: \"kubernetes.io/projected/09ec8375-f77f-4b8f-a21c-9303968468ca-kube-api-access-bvb5z\") pod \"payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv\" (UID: \"09ec8375-f77f-4b8f-a21c-9303968468ca\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"
Nov 02 09:44:39.591420 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.582235    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"
Nov 02 09:44:39.639266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.635503    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"
Nov 02 09:44:40.000628 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:39.999730    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"]
Nov 02 09:44:40.004590 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:40.004459    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"]
Nov 02 09:44:40.255294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:40.254820    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:40.255294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:40.254884    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:40.586426 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:40.586387    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n" event={"ID":"7f72acb2-73e1-4e88-8639-de0fb5ddce30","Type":"ContainerStarted","Data":"ea657f63575b49626dbe683db4b461926e65d695c5d11e105daec88cde15b106"}
Nov 02 09:44:40.586637 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:40.586492    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n" event={"ID":"7f72acb2-73e1-4e88-8639-de0fb5ddce30","Type":"ContainerStarted","Data":"8cfb5caae6ee540726c9c5eab1c5e4bf5445a2931ef96863a45d51fcc1675815"}
Nov 02 09:44:40.586637 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:40.586524    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"
Nov 02 09:44:40.589114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:40.588719    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv" event={"ID":"09ec8375-f77f-4b8f-a21c-9303968468ca","Type":"ContainerStarted","Data":"345b6f469a814342ef530b035b42b4dbf711cd303c3506d5bbd5ae894a0bcc1e"}
Nov 02 09:44:40.589114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:40.588942    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"
Nov 02 09:44:40.589114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:40.588972    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv" event={"ID":"09ec8375-f77f-4b8f-a21c-9303968468ca","Type":"ContainerStarted","Data":"0ebc2bc6e4759116045b906c04be0ab3449063c4879dc219f44e9c16f7d7000d"}
Nov 02 09:44:40.607697 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:40.607645    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n" podStartSLOduration=2.6076299069999997 podStartE2EDuration="2.607629907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:40.604945547 +0000 UTC m=+557.543232347" watchObservedRunningTime="2026-11-02 09:44:40.607629907 +0000 UTC m=+557.545916707"
Nov 02 09:44:40.624094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:40.624036    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv" podStartSLOduration=1.624023593 podStartE2EDuration="1.624023593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:40.623673763 +0000 UTC m=+557.561960566" watchObservedRunningTime="2026-11-02 09:44:40.624023593 +0000 UTC m=+557.562310369"
Nov 02 09:44:41.170495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:41.170386    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:41.170495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:41.170464    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:41.296491 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:41.296445    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:41.297187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:41.296526    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:41.601241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:41.601142    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z" event={"ID":"180a67b5-4edb-4572-b71c-5913c10a34e3","Type":"ContainerStarted","Data":"cb551ad31dae561e6989f82334b51c8ec0fc861ba2809c15d14aa2133ebc86b5"}
Nov 02 09:44:41.620832 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:41.620768    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z" podStartSLOduration=2.248867621 podStartE2EDuration="14.620753385s" totalImagesPullingTime="12.371885764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:41.619266732 +0000 UTC m=+558.557553532" watchObservedRunningTime="2026-11-02 09:44:41.620753385 +0000 UTC m=+558.559040160"
Nov 02 09:44:42.338415 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:42.338377    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:42.338962 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:42.338445    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:42.612029 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:42.611939    4449 generic.go:356] "Generic (PLEG): container finished" podID="180a67b5-4edb-4572-b71c-5913c10a34e3" containerID="cb551ad31dae561e6989f82334b51c8ec0fc861ba2809c15d14aa2133ebc86b5" exitCode=0
Nov 02 09:44:42.612197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:42.612029    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z" event={"ID":"180a67b5-4edb-4572-b71c-5913c10a34e3","Type":"ContainerDied","Data":"cb551ad31dae561e6989f82334b51c8ec0fc861ba2809c15d14aa2133ebc86b5"}
Nov 02 09:44:42.614149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:42.614115    4449 generic.go:356] "Generic (PLEG): container finished" podID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerID="eceb51ab9f2e960822f1c4ac856d6ee28714fa0e15601f108be40d068a7ba3cb" exitCode=0
Nov 02 09:44:42.614273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:42.614166    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" event={"ID":"bcfda7a4-ba6a-4941-9b13-a79d554fa88a","Type":"ContainerDied","Data":"eceb51ab9f2e960822f1c4ac856d6ee28714fa0e15601f108be40d068a7ba3cb"}
Nov 02 09:44:42.614664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:42.614645    4449 scope.go:122] "RemoveContainer" containerID="eceb51ab9f2e960822f1c4ac856d6ee28714fa0e15601f108be40d068a7ba3cb"
Nov 02 09:44:42.616284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:42.616245    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:42.616360 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:42.616331    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:42.616409 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:42.616382    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:42.691171 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:42.691138    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:42.692759 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:42.692736    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92"
Nov 02 09:44:43.622316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:43.622277    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" event={"ID":"bcfda7a4-ba6a-4941-9b13-a79d554fa88a","Type":"ContainerStarted","Data":"edb6b950f6cf882054c331f8062ef962daffd73a3554403e7b35df3addf5ff75"}
Nov 02 09:44:43.809642 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:43.809618    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"
Nov 02 09:44:43.911286 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:43.911167    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-openshift-ca\") pod \"180a67b5-4edb-4572-b71c-5913c10a34e3\" (UID: \"180a67b5-4edb-4572-b71c-5913c10a34e3\") "
Nov 02 09:44:43.911286 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:43.911272    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/180a67b5-4edb-4572-b71c-5913c10a34e3-kube-api-access-4kt7r\" (UniqueName: \"kubernetes.io/projected/180a67b5-4edb-4572-b71c-5913c10a34e3-kube-api-access-4kt7r\") pod \"180a67b5-4edb-4572-b71c-5913c10a34e3\" (UID: \"180a67b5-4edb-4572-b71c-5913c10a34e3\") "
Nov 02 09:44:43.911547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:43.911522    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-xks-ca\") pod \"180a67b5-4edb-4572-b71c-5913c10a34e3\" (UID: \"180a67b5-4edb-4572-b71c-5913c10a34e3\") "
Nov 02 09:44:43.911638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:43.911611    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-openshift-ca" pod "180a67b5-4edb-4572-b71c-5913c10a34e3" (UID: "180a67b5-4edb-4572-b71c-5913c10a34e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:44:43.911971 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:43.911951    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:44:43.913777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:43.913745    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-xks-ca" pod "180a67b5-4edb-4572-b71c-5913c10a34e3" (UID: "180a67b5-4edb-4572-b71c-5913c10a34e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:44:43.913882 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:43.913812    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180a67b5-4edb-4572-b71c-5913c10a34e3-kube-api-access-4kt7r" pod "180a67b5-4edb-4572-b71c-5913c10a34e3" (UID: "180a67b5-4edb-4572-b71c-5913c10a34e3"). InnerVolumeSpecName "kube-api-access-4kt7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:44:44.013706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:44.013668    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kt7r\" (UniqueName: \"kubernetes.io/projected/180a67b5-4edb-4572-b71c-5913c10a34e3-kube-api-access-4kt7r\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:44:44.013706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:44.013705    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180a67b5-4edb-4572-b71c-5913c10a34e3-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:44:44.585312 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:44.585279    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:44.585491 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:44.585472    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 09:44:44.585542 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:44.585520    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 09:44:44.629920 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:44.629890    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"
Nov 02 09:44:44.630320 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:44.629933    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z" event={"ID":"180a67b5-4edb-4572-b71c-5913c10a34e3","Type":"ContainerDied","Data":"6a6d9e10472125d7b36cba9ea19b89c07434c612576db289680590645ab297b2"}
Nov 02 09:44:44.630320 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:44.629988    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a6d9e10472125d7b36cba9ea19b89c07434c612576db289680590645ab297b2"
Nov 02 09:44:45.584726 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:45.584689    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 09:44:45.584905 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:45.584759    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 09:44:46.584533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:46.584498    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 09:44:46.584944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:46.584558    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 09:44:47.584769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:47.584734    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 09:44:47.585348 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:47.584793    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 09:44:48.584977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:48.584939    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 09:44:48.585669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:48.585633    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 09:44:49.588543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:49.588509    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:49.588997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:49.588926    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:49.589799 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:49.589779    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:44:51.604705 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:51.604676    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"
Nov 02 09:44:51.605119 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:51.604742    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"
Nov 02 09:44:51.965612 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:51.965532    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"]
Nov 02 09:44:51.966518 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:51.966490    4449 cpu_manager.go:414] "containerMap: removing container" podUID="180a67b5-4edb-4572-b71c-5913c10a34e3" containerName="revoke-keys"
Nov 02 09:44:51.966518 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:51.966519    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180a67b5-4edb-4572-b71c-5913c10a34e3" containerName="revoke-keys"
Nov 02 09:44:51.966721 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:51.966679    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="180a67b5-4edb-4572-b71c-5913c10a34e3" containerName="revoke-keys"
Nov 02 09:44:51.973481 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:51.973462    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:51.978603 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:51.978569    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-bde944-openshift-default-dockercfg-hb9wq\""
Nov 02 09:44:51.980649 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:51.980623    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"]
Nov 02 09:44:52.059498 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.059455    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.059498 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.059494    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.059772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.059526    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.059772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.059672    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.059772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.059714    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.059944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.059822    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.059944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.059877    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.059944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.059930    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9csh\" (UniqueName: \"kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-kube-api-access-f9csh\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.060100 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.059971    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.160580 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.160543    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.160795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.160592    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.160795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.160624    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.160795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.160655    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.160795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.160685    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.160795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.160709    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.160795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.160756    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.161124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.160801    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9csh\" (UniqueName: \"kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-kube-api-access-f9csh\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.161124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.160849    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.161124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.161041    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.161316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.161240    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.161407 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.161389    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.161467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.161420    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.161467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.161452    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.163404 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.163379    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.163504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.163442    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.173174 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.173144    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9csh\" (UniqueName: \"kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-kube-api-access-f9csh\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.173673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.173655    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.288056 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.288026    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:52.576390 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.576357    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"]
Nov 02 09:44:52.577022 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:44:52.576993    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab2b6c8_fd73_43f8_811c_d31f347557bf.slice/crio-99bcfa9b438a9004621bb1768f71d12874879885709d47c43d18ba58cb8a3bc9 WatchSource:0}: Error finding container 99bcfa9b438a9004621bb1768f71d12874879885709d47c43d18ba58cb8a3bc9: Status 404 returned error can't find the container with id 99bcfa9b438a9004621bb1768f71d12874879885709d47c43d18ba58cb8a3bc9
Nov 02 09:44:52.579411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.579375    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:52.579507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.579445    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:52.579507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.579478    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:44:52.691023 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.690985    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" event={"ID":"2ab2b6c8-fd73-43f8-811c-d31f347557bf","Type":"ContainerStarted","Data":"f458b4a3b282515d2613beb5e7dc1d805f82b60b967d39b0ab0b937410cdeb00"}
Nov 02 09:44:52.691525 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.691044    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" event={"ID":"2ab2b6c8-fd73-43f8-811c-d31f347557bf","Type":"ContainerStarted","Data":"99bcfa9b438a9004621bb1768f71d12874879885709d47c43d18ba58cb8a3bc9"}
Nov 02 09:44:52.714403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:52.714351    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" podStartSLOduration=1.7143373830000002 podStartE2EDuration="1.714337383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:52.712330286 +0000 UTC m=+569.650617062" watchObservedRunningTime="2026-11-02 09:44:52.714337383 +0000 UTC m=+569.652624137"
Nov 02 09:44:53.288510 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:53.288477    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:53.293736 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:53.293712    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:53.697281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:53.697158    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:53.698276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:53.698260    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:44:56.418601 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.418563    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"]
Nov 02 09:44:56.421738 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.421720    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"
Nov 02 09:44:56.428789 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.428764    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-bde944\""
Nov 02 09:44:56.434322 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.434297    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"]
Nov 02 09:44:56.440351 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.440324    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"]
Nov 02 09:44:56.445136 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.445112    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"
Nov 02 09:44:56.453309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.453278    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"]
Nov 02 09:44:56.454443 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.454417    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-bde944-dockercfg-ljd7z\""
Nov 02 09:44:56.503959 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.503925    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7lr4\" (UniqueName: \"kubernetes.io/projected/7fcc4135-f9c4-4c04-96da-26ad2abd502e-kube-api-access-q7lr4\") pod \"payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t\" (UID: \"7fcc4135-f9c4-4c04-96da-26ad2abd502e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"
Nov 02 09:44:56.503959 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.503962    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d1f11e-bd29-4c15-af90-027c65c690fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk\" (UID: \"35d1f11e-bd29-4c15-af90-027c65c690fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"
Nov 02 09:44:56.504152 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.503982    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvntd\" (UniqueName: \"kubernetes.io/projected/35d1f11e-bd29-4c15-af90-027c65c690fa-kube-api-access-kvntd\") pod \"payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk\" (UID: \"35d1f11e-bd29-4c15-af90-027c65c690fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"
Nov 02 09:44:56.504205 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.504179    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcc4135-f9c4-4c04-96da-26ad2abd502e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t\" (UID: \"7fcc4135-f9c4-4c04-96da-26ad2abd502e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"
Nov 02 09:44:56.605494 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.605447    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7lr4\" (UniqueName: \"kubernetes.io/projected/7fcc4135-f9c4-4c04-96da-26ad2abd502e-kube-api-access-q7lr4\") pod \"payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t\" (UID: \"7fcc4135-f9c4-4c04-96da-26ad2abd502e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"
Nov 02 09:44:56.605661 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.605512    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d1f11e-bd29-4c15-af90-027c65c690fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk\" (UID: \"35d1f11e-bd29-4c15-af90-027c65c690fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"
Nov 02 09:44:56.605661 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.605542    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvntd\" (UniqueName: \"kubernetes.io/projected/35d1f11e-bd29-4c15-af90-027c65c690fa-kube-api-access-kvntd\") pod \"payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk\" (UID: \"35d1f11e-bd29-4c15-af90-027c65c690fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"
Nov 02 09:44:56.605661 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.605648    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcc4135-f9c4-4c04-96da-26ad2abd502e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t\" (UID: \"7fcc4135-f9c4-4c04-96da-26ad2abd502e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"
Nov 02 09:44:56.606899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.606598    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d1f11e-bd29-4c15-af90-027c65c690fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk\" (UID: \"35d1f11e-bd29-4c15-af90-027c65c690fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"
Nov 02 09:44:56.606899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.606781    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcc4135-f9c4-4c04-96da-26ad2abd502e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t\" (UID: \"7fcc4135-f9c4-4c04-96da-26ad2abd502e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"
Nov 02 09:44:56.618386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.618354    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7lr4\" (UniqueName: \"kubernetes.io/projected/7fcc4135-f9c4-4c04-96da-26ad2abd502e-kube-api-access-q7lr4\") pod \"payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t\" (UID: \"7fcc4135-f9c4-4c04-96da-26ad2abd502e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"
Nov 02 09:44:56.618529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.618472    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvntd\" (UniqueName: \"kubernetes.io/projected/35d1f11e-bd29-4c15-af90-027c65c690fa-kube-api-access-kvntd\") pod \"payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk\" (UID: \"35d1f11e-bd29-4c15-af90-027c65c690fa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"
Nov 02 09:44:56.738149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.738075    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"
Nov 02 09:44:56.782437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:56.782397    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"
Nov 02 09:44:57.081771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:57.081739    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"]
Nov 02 09:44:57.082032 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:44:57.082009    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcc4135_f9c4_4c04_96da_26ad2abd502e.slice/crio-a7418bbe5552f0c37a05887a2ff121ca017de86a707340c9808d8256b81499f9 WatchSource:0}: Error finding container a7418bbe5552f0c37a05887a2ff121ca017de86a707340c9808d8256b81499f9: Status 404 returned error can't find the container with id a7418bbe5552f0c37a05887a2ff121ca017de86a707340c9808d8256b81499f9
Nov 02 09:44:57.113481 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:57.113448    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"]
Nov 02 09:44:57.785499 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:57.785448    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t" event={"ID":"7fcc4135-f9c4-4c04-96da-26ad2abd502e","Type":"ContainerStarted","Data":"3156aa548a2904e0ade7d04c4d901bc6de93f99bc34a66c365da3a0b07de8673"}
Nov 02 09:44:57.801924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:57.801885    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"
Nov 02 09:44:57.803321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:57.803278    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t" event={"ID":"7fcc4135-f9c4-4c04-96da-26ad2abd502e","Type":"ContainerStarted","Data":"a7418bbe5552f0c37a05887a2ff121ca017de86a707340c9808d8256b81499f9"}
Nov 02 09:44:57.818241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:57.813756    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk" event={"ID":"35d1f11e-bd29-4c15-af90-027c65c690fa","Type":"ContainerStarted","Data":"e3dfe6a5133d381ba3b385b7243890b474b52df8d1898892013a1ea77d4d146d"}
Nov 02 09:44:57.818241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:57.813811    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk" event={"ID":"35d1f11e-bd29-4c15-af90-027c65c690fa","Type":"ContainerStarted","Data":"7ed5adf4d564e91038f9678ba1eb5cfaa4b04e491f0ab936b79ab57f92cecc99"}
Nov 02 09:44:57.818241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:57.814272    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"
Nov 02 09:44:57.853002 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:57.852928    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t" podStartSLOduration=1.852905899 podStartE2EDuration="1.852905899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:57.82361496 +0000 UTC m=+574.761901754" watchObservedRunningTime="2026-11-02 09:44:57.852905899 +0000 UTC m=+574.791192678"
Nov 02 09:44:57.887264 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:44:57.876630    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk" podStartSLOduration=1.876612594 podStartE2EDuration="1.876612594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:57.875620283 +0000 UTC m=+574.813907062" watchObservedRunningTime="2026-11-02 09:44:57.876612594 +0000 UTC m=+574.814899373"
Nov 02 09:45:00.150452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.150408    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"]
Nov 02 09:45:00.154693 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.154669    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"]
Nov 02 09:45:00.154840 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.154786    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"
Nov 02 09:45:00.158204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.158182    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"]
Nov 02 09:45:00.158318 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.158301    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"
Nov 02 09:45:00.162094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.162060    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"
Nov 02 09:45:00.164024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.164001    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"]
Nov 02 09:45:00.165171 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.165081    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:45:00.165327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.165310    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:45:00.165398 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.165373    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:45:00.171311 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.171272    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"]
Nov 02 09:45:00.172937 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.172912    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"]
Nov 02 09:45:00.296745 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.296712    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgp28\" (UniqueName: \"kubernetes.io/projected/059ffd84-6945-4f53-a912-7fe1328708e3-kube-api-access-fgp28\") pod \"maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c\" (UID: \"059ffd84-6945-4f53-a912-7fe1328708e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"
Nov 02 09:45:00.296923 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.296750    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ababc0-9768-4b79-a3e3-0c6f510aa287-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2\" (UID: \"f2ababc0-9768-4b79-a3e3-0c6f510aa287\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"
Nov 02 09:45:00.296923 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.296779    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5ft\" (UniqueName: \"kubernetes.io/projected/19995443-0610-41df-b899-dbc0ac2c0360-kube-api-access-6b5ft\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg\" (UID: \"19995443-0610-41df-b899-dbc0ac2c0360\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"
Nov 02 09:45:00.296923 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.296905    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059ffd84-6945-4f53-a912-7fe1328708e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c\" (UID: \"059ffd84-6945-4f53-a912-7fe1328708e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"
Nov 02 09:45:00.297047 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.296945    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19995443-0610-41df-b899-dbc0ac2c0360-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg\" (UID: \"19995443-0610-41df-b899-dbc0ac2c0360\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"
Nov 02 09:45:00.297047 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.296972    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5p9v\" (UniqueName: \"kubernetes.io/projected/f2ababc0-9768-4b79-a3e3-0c6f510aa287-kube-api-access-c5p9v\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2\" (UID: \"f2ababc0-9768-4b79-a3e3-0c6f510aa287\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"
Nov 02 09:45:00.398638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.398599    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgp28\" (UniqueName: \"kubernetes.io/projected/059ffd84-6945-4f53-a912-7fe1328708e3-kube-api-access-fgp28\") pod \"maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c\" (UID: \"059ffd84-6945-4f53-a912-7fe1328708e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"
Nov 02 09:45:00.398811 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.398647    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ababc0-9768-4b79-a3e3-0c6f510aa287-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2\" (UID: \"f2ababc0-9768-4b79-a3e3-0c6f510aa287\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"
Nov 02 09:45:00.398892 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.398823    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5ft\" (UniqueName: \"kubernetes.io/projected/19995443-0610-41df-b899-dbc0ac2c0360-kube-api-access-6b5ft\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg\" (UID: \"19995443-0610-41df-b899-dbc0ac2c0360\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"
Nov 02 09:45:00.398892 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.398863    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059ffd84-6945-4f53-a912-7fe1328708e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c\" (UID: \"059ffd84-6945-4f53-a912-7fe1328708e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"
Nov 02 09:45:00.398892 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.398891    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19995443-0610-41df-b899-dbc0ac2c0360-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg\" (UID: \"19995443-0610-41df-b899-dbc0ac2c0360\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"
Nov 02 09:45:00.399119 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.398917    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5p9v\" (UniqueName: \"kubernetes.io/projected/f2ababc0-9768-4b79-a3e3-0c6f510aa287-kube-api-access-c5p9v\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2\" (UID: \"f2ababc0-9768-4b79-a3e3-0c6f510aa287\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"
Nov 02 09:45:00.399457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.399434    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ababc0-9768-4b79-a3e3-0c6f510aa287-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2\" (UID: \"f2ababc0-9768-4b79-a3e3-0c6f510aa287\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"
Nov 02 09:45:00.399595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.399576    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059ffd84-6945-4f53-a912-7fe1328708e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c\" (UID: \"059ffd84-6945-4f53-a912-7fe1328708e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"
Nov 02 09:45:00.399701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.399679    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19995443-0610-41df-b899-dbc0ac2c0360-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg\" (UID: \"19995443-0610-41df-b899-dbc0ac2c0360\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"
Nov 02 09:45:00.411747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.411690    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5ft\" (UniqueName: \"kubernetes.io/projected/19995443-0610-41df-b899-dbc0ac2c0360-kube-api-access-6b5ft\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg\" (UID: \"19995443-0610-41df-b899-dbc0ac2c0360\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"
Nov 02 09:45:00.411859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.411783    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5p9v\" (UniqueName: \"kubernetes.io/projected/f2ababc0-9768-4b79-a3e3-0c6f510aa287-kube-api-access-c5p9v\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2\" (UID: \"f2ababc0-9768-4b79-a3e3-0c6f510aa287\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"
Nov 02 09:45:00.411923 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.411889    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgp28\" (UniqueName: \"kubernetes.io/projected/059ffd84-6945-4f53-a912-7fe1328708e3-kube-api-access-fgp28\") pod \"maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c\" (UID: \"059ffd84-6945-4f53-a912-7fe1328708e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"
Nov 02 09:45:00.479911 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.479880    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"
Nov 02 09:45:00.500079 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.500048    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"
Nov 02 09:45:00.502166 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.502145    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"
Nov 02 09:45:00.765993 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.765959    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"]
Nov 02 09:45:00.767161 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:45:00.767136    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19995443_0610_41df_b899_dbc0ac2c0360.slice/crio-50bf9034a2b5b98586d686388d9527054ac5f23d144806e9c93b5e9d35b1e711 WatchSource:0}: Error finding container 50bf9034a2b5b98586d686388d9527054ac5f23d144806e9c93b5e9d35b1e711: Status 404 returned error can't find the container with id 50bf9034a2b5b98586d686388d9527054ac5f23d144806e9c93b5e9d35b1e711
Nov 02 09:45:00.817123 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.817091    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"]
Nov 02 09:45:00.818189 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:45:00.818149    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059ffd84_6945_4f53_a912_7fe1328708e3.slice/crio-5ef66ea43598a81337b3fe7ee73171374f4bbf57d3ec74da3219d31663ea0c45 WatchSource:0}: Error finding container 5ef66ea43598a81337b3fe7ee73171374f4bbf57d3ec74da3219d31663ea0c45: Status 404 returned error can't find the container with id 5ef66ea43598a81337b3fe7ee73171374f4bbf57d3ec74da3219d31663ea0c45
Nov 02 09:45:00.820958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.820929    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"]
Nov 02 09:45:00.821849 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:45:00.821829    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ababc0_9768_4b79_a3e3_0c6f510aa287.slice/crio-9b26fb84075b0262cb967b009bd9724b8a9b9120821896713e6ff285c4f20908 WatchSource:0}: Error finding container 9b26fb84075b0262cb967b009bd9724b8a9b9120821896713e6ff285c4f20908: Status 404 returned error can't find the container with id 9b26fb84075b0262cb967b009bd9724b8a9b9120821896713e6ff285c4f20908
Nov 02 09:45:00.834806 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.834776    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c" event={"ID":"059ffd84-6945-4f53-a912-7fe1328708e3","Type":"ContainerStarted","Data":"5ef66ea43598a81337b3fe7ee73171374f4bbf57d3ec74da3219d31663ea0c45"}
Nov 02 09:45:00.836201 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.836173    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg" event={"ID":"19995443-0610-41df-b899-dbc0ac2c0360","Type":"ContainerStarted","Data":"50bf9034a2b5b98586d686388d9527054ac5f23d144806e9c93b5e9d35b1e711"}
Nov 02 09:45:00.837710 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:00.837685    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2" event={"ID":"f2ababc0-9768-4b79-a3e3-0c6f510aa287","Type":"ContainerStarted","Data":"9b26fb84075b0262cb967b009bd9724b8a9b9120821896713e6ff285c4f20908"}
Nov 02 09:45:01.846176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:01.846065    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c" event={"ID":"059ffd84-6945-4f53-a912-7fe1328708e3","Type":"ContainerStarted","Data":"5878761baffd33f66ef0b475dd1f12e81a3a5579c15f7664994ba0312d24835a"}
Nov 02 09:45:01.847766 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:01.847734    4449 generic.go:356] "Generic (PLEG): container finished" podID="19995443-0610-41df-b899-dbc0ac2c0360" containerID="f55680a0de867ace711f0768b96824efdb409440f0a58afa7c96e46050456ada" exitCode=7
Nov 02 09:45:01.847904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:01.847769    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg" event={"ID":"19995443-0610-41df-b899-dbc0ac2c0360","Type":"ContainerDied","Data":"f55680a0de867ace711f0768b96824efdb409440f0a58afa7c96e46050456ada"}
Nov 02 09:45:01.848081 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:01.848063    4449 scope.go:122] "RemoveContainer" containerID="f55680a0de867ace711f0768b96824efdb409440f0a58afa7c96e46050456ada"
Nov 02 09:45:01.849554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:01.849525    4449 generic.go:356] "Generic (PLEG): container finished" podID="f2ababc0-9768-4b79-a3e3-0c6f510aa287" containerID="236de639319b012155a90d1efadf746637e68c292de302f10abf2577d5bc97d9" exitCode=60
Nov 02 09:45:01.849657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:01.849565    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2" event={"ID":"f2ababc0-9768-4b79-a3e3-0c6f510aa287","Type":"ContainerDied","Data":"236de639319b012155a90d1efadf746637e68c292de302f10abf2577d5bc97d9"}
Nov 02 09:45:01.849838 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:01.849822    4449 scope.go:122] "RemoveContainer" containerID="236de639319b012155a90d1efadf746637e68c292de302f10abf2577d5bc97d9"
Nov 02 09:45:01.863904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:01.863862    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c" podStartSLOduration=1.863850708 podStartE2EDuration="1.863850708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:01.862683751 +0000 UTC m=+578.800970532" watchObservedRunningTime="2026-11-02 09:45:01.863850708 +0000 UTC m=+578.802137501"
Nov 02 09:45:02.857787 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:02.857754    4449 generic.go:356] "Generic (PLEG): container finished" podID="19995443-0610-41df-b899-dbc0ac2c0360" containerID="5e4bd172169dde10e8d91a630c3d8aa122a5712bcd09ebf259c4a36f280d5d49" exitCode=7
Nov 02 09:45:02.858153 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:02.857826    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg" event={"ID":"19995443-0610-41df-b899-dbc0ac2c0360","Type":"ContainerDied","Data":"5e4bd172169dde10e8d91a630c3d8aa122a5712bcd09ebf259c4a36f280d5d49"}
Nov 02 09:45:02.858153 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:02.857890    4449 scope.go:122] "RemoveContainer" containerID="f55680a0de867ace711f0768b96824efdb409440f0a58afa7c96e46050456ada"
Nov 02 09:45:02.858153 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:02.858133    4449 scope.go:122] "RemoveContainer" containerID="5e4bd172169dde10e8d91a630c3d8aa122a5712bcd09ebf259c4a36f280d5d49"
Nov 02 09:45:02.858464 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:45:02.858438    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg_odh-ai-gateway-infra(19995443-0610-41df-b899-dbc0ac2c0360)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg" podUID="19995443-0610-41df-b899-dbc0ac2c0360"
Nov 02 09:45:02.859923 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:02.859900    4449 generic.go:356] "Generic (PLEG): container finished" podID="f2ababc0-9768-4b79-a3e3-0c6f510aa287" containerID="9e46d7446249ce86f59592ce0d82cbe1c7ab56630071cf161da3ee7d3124774c" exitCode=60
Nov 02 09:45:02.860032 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:02.859984    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2" event={"ID":"f2ababc0-9768-4b79-a3e3-0c6f510aa287","Type":"ContainerDied","Data":"9e46d7446249ce86f59592ce0d82cbe1c7ab56630071cf161da3ee7d3124774c"}
Nov 02 09:45:02.860270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:02.860253    4449 scope.go:122] "RemoveContainer" containerID="9e46d7446249ce86f59592ce0d82cbe1c7ab56630071cf161da3ee7d3124774c"
Nov 02 09:45:02.860524 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:45:02.860506    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2_odh-ai-gateway-infra(f2ababc0-9768-4b79-a3e3-0c6f510aa287)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2" podUID="f2ababc0-9768-4b79-a3e3-0c6f510aa287"
Nov 02 09:45:02.861680 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:02.861659    4449 generic.go:356] "Generic (PLEG): container finished" podID="059ffd84-6945-4f53-a912-7fe1328708e3" containerID="5878761baffd33f66ef0b475dd1f12e81a3a5579c15f7664994ba0312d24835a" exitCode=60
Nov 02 09:45:02.861782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:02.861733    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c" event={"ID":"059ffd84-6945-4f53-a912-7fe1328708e3","Type":"ContainerDied","Data":"5878761baffd33f66ef0b475dd1f12e81a3a5579c15f7664994ba0312d24835a"}
Nov 02 09:45:02.861916 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:02.861897    4449 scope.go:122] "RemoveContainer" containerID="5878761baffd33f66ef0b475dd1f12e81a3a5579c15f7664994ba0312d24835a"
Nov 02 09:45:02.922611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:02.922589    4449 scope.go:122] "RemoveContainer" containerID="236de639319b012155a90d1efadf746637e68c292de302f10abf2577d5bc97d9"
Nov 02 09:45:03.884299 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:03.884256    4449 scope.go:122] "RemoveContainer" containerID="5e4bd172169dde10e8d91a630c3d8aa122a5712bcd09ebf259c4a36f280d5d49"
Nov 02 09:45:03.884768 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:45:03.884600    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg_odh-ai-gateway-infra(19995443-0610-41df-b899-dbc0ac2c0360)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg" podUID="19995443-0610-41df-b899-dbc0ac2c0360"
Nov 02 09:45:03.886452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:03.886430    4449 scope.go:122] "RemoveContainer" containerID="9e46d7446249ce86f59592ce0d82cbe1c7ab56630071cf161da3ee7d3124774c"
Nov 02 09:45:03.886735 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:45:03.886700    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2_odh-ai-gateway-infra(f2ababc0-9768-4b79-a3e3-0c6f510aa287)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2" podUID="f2ababc0-9768-4b79-a3e3-0c6f510aa287"
Nov 02 09:45:03.888628 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:03.888600    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c" event={"ID":"059ffd84-6945-4f53-a912-7fe1328708e3","Type":"ContainerStarted","Data":"f5819896937a4e1fe9ac2e205ad8a8ab0d54a37d54d660b755030e5400b79047"}
Nov 02 09:45:04.897953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:04.897867    4449 generic.go:356] "Generic (PLEG): container finished" podID="059ffd84-6945-4f53-a912-7fe1328708e3" containerID="f5819896937a4e1fe9ac2e205ad8a8ab0d54a37d54d660b755030e5400b79047" exitCode=60
Nov 02 09:45:04.898423 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:04.897951    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c" event={"ID":"059ffd84-6945-4f53-a912-7fe1328708e3","Type":"ContainerDied","Data":"f5819896937a4e1fe9ac2e205ad8a8ab0d54a37d54d660b755030e5400b79047"}
Nov 02 09:45:04.898423 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:04.898006    4449 scope.go:122] "RemoveContainer" containerID="5878761baffd33f66ef0b475dd1f12e81a3a5579c15f7664994ba0312d24835a"
Nov 02 09:45:04.898423 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:04.898293    4449 scope.go:122] "RemoveContainer" containerID="f5819896937a4e1fe9ac2e205ad8a8ab0d54a37d54d660b755030e5400b79047"
Nov 02 09:45:04.898573 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:45:04.898546    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c_odh-ai-gateway-infra(059ffd84-6945-4f53-a912-7fe1328708e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c" podUID="059ffd84-6945-4f53-a912-7fe1328708e3"
Nov 02 09:45:05.906594 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:05.906562    4449 scope.go:122] "RemoveContainer" containerID="f5819896937a4e1fe9ac2e205ad8a8ab0d54a37d54d660b755030e5400b79047"
Nov 02 09:45:05.906956 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:45:05.906815    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c_odh-ai-gateway-infra(059ffd84-6945-4f53-a912-7fe1328708e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c" podUID="059ffd84-6945-4f53-a912-7fe1328708e3"
Nov 02 09:45:06.546135 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:45:06.546056    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:45:08.825916 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:08.825877    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"
Nov 02 09:45:09.829331 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:09.829298    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"
Nov 02 09:45:11.022105 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.022064    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"]
Nov 02 09:45:11.030486 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.030459    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.042566 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.042542    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-9e7156-openshift-default-dockercfg-nlp5g\""
Nov 02 09:45:11.049587 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.049559    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"]
Nov 02 09:45:11.057111 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.057078    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.057307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.057134    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjt2p\" (UniqueName: \"kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-kube-api-access-vjt2p\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.057307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.057274    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.057405 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.057345    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.057481 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.057457    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.057553 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.057540    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.057708 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.057680    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.057834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.057744    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.057834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.057792    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c312869-83d6-406b-bff4-6dc3068df5bf-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.162380    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.162459    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.162529    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.162604    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.162660    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.162703    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.162748    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c312869-83d6-406b-bff4-6dc3068df5bf-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.162814    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.162870    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjt2p\" (UniqueName: \"kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-kube-api-access-vjt2p\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.163736    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.163984    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.165378    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c312869-83d6-406b-bff4-6dc3068df5bf-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.165679    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.174242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.165900    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.177747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.177713    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.179945 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.179918    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.181575 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.181157    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjt2p\" (UniqueName: \"kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-kube-api-access-vjt2p\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.183197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.182923    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.346707 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.346670    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:11.642467 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:45:11.642431    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c312869_83d6_406b_bff4_6dc3068df5bf.slice/crio-21b9bdb288c6f7f9d516f8a3b68c72e2a7df7cfac5e8599261cf6e10e16afdaf WatchSource:0}: Error finding container 21b9bdb288c6f7f9d516f8a3b68c72e2a7df7cfac5e8599261cf6e10e16afdaf: Status 404 returned error can't find the container with id 21b9bdb288c6f7f9d516f8a3b68c72e2a7df7cfac5e8599261cf6e10e16afdaf
Nov 02 09:45:11.642615 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.642536    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"]
Nov 02 09:45:11.647366 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.647322    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:45:11.647496 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.647416    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:45:11.647496 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.647460    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:45:11.957008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.956914    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" event={"ID":"0c312869-83d6-406b-bff4-6dc3068df5bf","Type":"ContainerStarted","Data":"fc47298d4dbb14070fd07471ea1613b082cf73f8b47a7ced1dbf45924739a10d"}
Nov 02 09:45:11.957008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.956973    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" event={"ID":"0c312869-83d6-406b-bff4-6dc3068df5bf","Type":"ContainerStarted","Data":"21b9bdb288c6f7f9d516f8a3b68c72e2a7df7cfac5e8599261cf6e10e16afdaf"}
Nov 02 09:45:11.989790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:11.989732    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" podStartSLOduration=1.9897163610000002 podStartE2EDuration="1.989716361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:11.987366053 +0000 UTC m=+588.925652870" watchObservedRunningTime="2026-11-02 09:45:11.989716361 +0000 UTC m=+588.928003136"
Nov 02 09:45:12.347069 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:12.347034    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:12.352504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:12.352474    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:12.963686 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:12.963655    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:12.965856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:12.964715    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:13.971377 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:13.971347    4449 generic.go:356] "Generic (PLEG): container finished" podID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerID="f458b4a3b282515d2613beb5e7dc1d805f82b60b967d39b0ab0b937410cdeb00" exitCode=0
Nov 02 09:45:13.971788 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:13.971422    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" event={"ID":"2ab2b6c8-fd73-43f8-811c-d31f347557bf","Type":"ContainerDied","Data":"f458b4a3b282515d2613beb5e7dc1d805f82b60b967d39b0ab0b937410cdeb00"}
Nov 02 09:45:13.971881 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:13.971865    4449 scope.go:122] "RemoveContainer" containerID="f458b4a3b282515d2613beb5e7dc1d805f82b60b967d39b0ab0b937410cdeb00"
Nov 02 09:45:13.973357 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:13.973329    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:45:13.973436 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:13.973397    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:45:13.973473 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:13.973432    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:45:14.199617 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:14.199588    4449 scope.go:122] "RemoveContainer" containerID="9e46d7446249ce86f59592ce0d82cbe1c7ab56630071cf161da3ee7d3124774c"
Nov 02 09:45:14.990015 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:14.989981    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" event={"ID":"2ab2b6c8-fd73-43f8-811c-d31f347557bf","Type":"ContainerStarted","Data":"62db54214e8204fa57c4e62a8114ba135017de1f898ba9d40061ec25d44fb52d"}
Nov 02 09:45:14.993039 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:14.993001    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2" event={"ID":"f2ababc0-9768-4b79-a3e3-0c6f510aa287","Type":"ContainerStarted","Data":"c85081b7e772f6d17e9b12f48de708f6f61eb9b897375de5323c5106128c8d6c"}
Nov 02 09:45:15.072469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.072059    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2" podStartSLOduration=15.072038517 podStartE2EDuration="15.072038517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:15.063138363 +0000 UTC m=+592.001425131" watchObservedRunningTime="2026-11-02 09:45:15.072038517 +0000 UTC m=+592.010325294"
Nov 02 09:45:15.289235 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.288645    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:45:15.289235 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.288776    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 09:45:15.289235 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.288820    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 09:45:15.824740 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.824705    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"]
Nov 02 09:45:15.834811 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.834776    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"]
Nov 02 09:45:15.836784 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.836748    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"
Nov 02 09:45:15.841204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.840374    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"]
Nov 02 09:45:15.841204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.840456    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"
Nov 02 09:45:15.844992 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.844969    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"]
Nov 02 09:45:15.849241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.849195    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-9e7156\""
Nov 02 09:45:15.933323 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.933276    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp\" (UID: \"c419bca8-6a23-4b7d-bfa1-c980a92c2e08\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"
Nov 02 09:45:15.934236 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.934187    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chrtq\" (UniqueName: \"kubernetes.io/projected/b57143d3-3a7a-4aa0-84e5-09e0b155a963-kube-api-access-chrtq\") pod \"payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds\" (UID: \"b57143d3-3a7a-4aa0-84e5-09e0b155a963\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"
Nov 02 09:45:15.934521 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.934498    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87gbm\" (UniqueName: \"kubernetes.io/projected/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-kube-api-access-87gbm\") pod \"payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp\" (UID: \"c419bca8-6a23-4b7d-bfa1-c980a92c2e08\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"
Nov 02 09:45:15.934741 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:15.934723    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57143d3-3a7a-4aa0-84e5-09e0b155a963-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds\" (UID: \"b57143d3-3a7a-4aa0-84e5-09e0b155a963\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"
Nov 02 09:45:16.036496 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.036458    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57143d3-3a7a-4aa0-84e5-09e0b155a963-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds\" (UID: \"b57143d3-3a7a-4aa0-84e5-09e0b155a963\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"
Nov 02 09:45:16.037230 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.037185    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp\" (UID: \"c419bca8-6a23-4b7d-bfa1-c980a92c2e08\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"
Nov 02 09:45:16.037401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.037385    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chrtq\" (UniqueName: \"kubernetes.io/projected/b57143d3-3a7a-4aa0-84e5-09e0b155a963-kube-api-access-chrtq\") pod \"payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds\" (UID: \"b57143d3-3a7a-4aa0-84e5-09e0b155a963\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"
Nov 02 09:45:16.037548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.037534    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87gbm\" (UniqueName: \"kubernetes.io/projected/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-kube-api-access-87gbm\") pod \"payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp\" (UID: \"c419bca8-6a23-4b7d-bfa1-c980a92c2e08\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"
Nov 02 09:45:16.038769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.038745    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57143d3-3a7a-4aa0-84e5-09e0b155a963-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds\" (UID: \"b57143d3-3a7a-4aa0-84e5-09e0b155a963\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"
Nov 02 09:45:16.040042 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.039972    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp\" (UID: \"c419bca8-6a23-4b7d-bfa1-c980a92c2e08\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"
Nov 02 09:45:16.042156 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.042128    4449 generic.go:356] "Generic (PLEG): container finished" podID="f2ababc0-9768-4b79-a3e3-0c6f510aa287" containerID="c85081b7e772f6d17e9b12f48de708f6f61eb9b897375de5323c5106128c8d6c" exitCode=60
Nov 02 09:45:16.044248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.043157    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2" event={"ID":"f2ababc0-9768-4b79-a3e3-0c6f510aa287","Type":"ContainerDied","Data":"c85081b7e772f6d17e9b12f48de708f6f61eb9b897375de5323c5106128c8d6c"}
Nov 02 09:45:16.044558 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.044540    4449 scope.go:122] "RemoveContainer" containerID="9e46d7446249ce86f59592ce0d82cbe1c7ab56630071cf161da3ee7d3124774c"
Nov 02 09:45:16.045382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.045366    4449 scope.go:122] "RemoveContainer" containerID="c85081b7e772f6d17e9b12f48de708f6f61eb9b897375de5323c5106128c8d6c"
Nov 02 09:45:16.048530 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:45:16.048507    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2_odh-ai-gateway-infra(f2ababc0-9768-4b79-a3e3-0c6f510aa287)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2" podUID="f2ababc0-9768-4b79-a3e3-0c6f510aa287"
Nov 02 09:45:16.101297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.073532    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"]
Nov 02 09:45:16.101297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.085919    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87gbm\" (UniqueName: \"kubernetes.io/projected/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-kube-api-access-87gbm\") pod \"payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp\" (UID: \"c419bca8-6a23-4b7d-bfa1-c980a92c2e08\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"
Nov 02 09:45:16.102355 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.102319    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chrtq\" (UniqueName: \"kubernetes.io/projected/b57143d3-3a7a-4aa0-84e5-09e0b155a963-kube-api-access-chrtq\") pod \"payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds\" (UID: \"b57143d3-3a7a-4aa0-84e5-09e0b155a963\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"
Nov 02 09:45:16.208586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.208548    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"
Nov 02 09:45:16.244446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.244421    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"
Nov 02 09:45:16.311230 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.311091    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 09:45:16.311230 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.311177    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 09:45:16.583051 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:45:16.583005    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc419bca8_6a23_4b7d_bfa1_c980a92c2e08.slice/crio-498847fcc56c8efa778c492b6e7d71571f495761ab8ff2179c72e2e7a7e7e5a8 WatchSource:0}: Error finding container 498847fcc56c8efa778c492b6e7d71571f495761ab8ff2179c72e2e7a7e7e5a8: Status 404 returned error can't find the container with id 498847fcc56c8efa778c492b6e7d71571f495761ab8ff2179c72e2e7a7e7e5a8
Nov 02 09:45:16.585426 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.585329    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"]
Nov 02 09:45:16.682174 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:16.681939    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"]
Nov 02 09:45:16.684896 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:45:16.684865    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57143d3_3a7a_4aa0_84e5_09e0b155a963.slice/crio-8a2d76c25984d0d4bd4136e4c45cbb28f965b594ea37ef9b9786956b9a8979ef WatchSource:0}: Error finding container 8a2d76c25984d0d4bd4136e4c45cbb28f965b594ea37ef9b9786956b9a8979ef: Status 404 returned error can't find the container with id 8a2d76c25984d0d4bd4136e4c45cbb28f965b594ea37ef9b9786956b9a8979ef
Nov 02 09:45:17.075730 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.073805    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds" event={"ID":"b57143d3-3a7a-4aa0-84e5-09e0b155a963","Type":"ContainerStarted","Data":"41b7b48494329bf7d1e9b3a666a856a1077cf0876c94bb6064d071d0a77781a1"}
Nov 02 09:45:17.075730 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.073876    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds" event={"ID":"b57143d3-3a7a-4aa0-84e5-09e0b155a963","Type":"ContainerStarted","Data":"8a2d76c25984d0d4bd4136e4c45cbb28f965b594ea37ef9b9786956b9a8979ef"}
Nov 02 09:45:17.075730 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.074822    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"
Nov 02 09:45:17.081183 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.081124    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp" event={"ID":"c419bca8-6a23-4b7d-bfa1-c980a92c2e08","Type":"ContainerStarted","Data":"8f459152acbd27aea887ab3a5637230d25782dd6051a6c5ca6b80ba7d8f09a0f"}
Nov 02 09:45:17.081183 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.081175    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp" event={"ID":"c419bca8-6a23-4b7d-bfa1-c980a92c2e08","Type":"ContainerStarted","Data":"498847fcc56c8efa778c492b6e7d71571f495761ab8ff2179c72e2e7a7e7e5a8"}
Nov 02 09:45:17.081356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.081258    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"
Nov 02 09:45:17.098934 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.098883    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds" podStartSLOduration=2.098867118 podStartE2EDuration="2.098867118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:17.095760426 +0000 UTC m=+594.034047204" watchObservedRunningTime="2026-11-02 09:45:17.098867118 +0000 UTC m=+594.037153941"
Nov 02 09:45:17.114954 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.114873    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp" podStartSLOduration=2.11485718 podStartE2EDuration="2.11485718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:17.114173394 +0000 UTC m=+594.052460189" watchObservedRunningTime="2026-11-02 09:45:17.11485718 +0000 UTC m=+594.053143956"
Nov 02 09:45:17.202611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.202583    4449 scope.go:122] "RemoveContainer" containerID="5e4bd172169dde10e8d91a630c3d8aa122a5712bcd09ebf259c4a36f280d5d49"
Nov 02 09:45:17.203297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.203273    4449 scope.go:122] "RemoveContainer" containerID="f5819896937a4e1fe9ac2e205ad8a8ab0d54a37d54d660b755030e5400b79047"
Nov 02 09:45:17.289280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.288420    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 09:45:17.289280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.288480    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 09:45:17.294006 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.293982    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"
Nov 02 09:45:17.385305 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.385194    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ababc0-9768-4b79-a3e3-0c6f510aa287-kube-api-access-c5p9v\" (UniqueName: \"kubernetes.io/projected/f2ababc0-9768-4b79-a3e3-0c6f510aa287-kube-api-access-c5p9v\") pod \"f2ababc0-9768-4b79-a3e3-0c6f510aa287\" (UID: \"f2ababc0-9768-4b79-a3e3-0c6f510aa287\") "
Nov 02 09:45:17.385305 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.385268    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ababc0-9768-4b79-a3e3-0c6f510aa287-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ababc0-9768-4b79-a3e3-0c6f510aa287-maas-api-service-ca\") pod \"f2ababc0-9768-4b79-a3e3-0c6f510aa287\" (UID: \"f2ababc0-9768-4b79-a3e3-0c6f510aa287\") "
Nov 02 09:45:17.385791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.385768    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ababc0-9768-4b79-a3e3-0c6f510aa287-maas-api-service-ca" pod "f2ababc0-9768-4b79-a3e3-0c6f510aa287" (UID: "f2ababc0-9768-4b79-a3e3-0c6f510aa287"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:45:17.387654 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.387624    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ababc0-9768-4b79-a3e3-0c6f510aa287-kube-api-access-c5p9v" pod "f2ababc0-9768-4b79-a3e3-0c6f510aa287" (UID: "f2ababc0-9768-4b79-a3e3-0c6f510aa287"). InnerVolumeSpecName "kube-api-access-c5p9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:45:17.487043 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.487012    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5p9v\" (UniqueName: \"kubernetes.io/projected/f2ababc0-9768-4b79-a3e3-0c6f510aa287-kube-api-access-c5p9v\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:17.487043 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:17.487039    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ababc0-9768-4b79-a3e3-0c6f510aa287-maas-api-service-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:18.088931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:18.088898    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg" event={"ID":"19995443-0610-41df-b899-dbc0ac2c0360","Type":"ContainerStarted","Data":"1a45676617fd059f462589534ac18a436a7de32919ceb4115d08bf18a798e1ff"}
Nov 02 09:45:18.090326 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:18.090307    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"
Nov 02 09:45:18.090463 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:18.090363    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2" event={"ID":"f2ababc0-9768-4b79-a3e3-0c6f510aa287","Type":"ContainerDied","Data":"9b26fb84075b0262cb967b009bd9724b8a9b9120821896713e6ff285c4f20908"}
Nov 02 09:45:18.090463 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:18.090405    4449 scope.go:122] "RemoveContainer" containerID="c85081b7e772f6d17e9b12f48de708f6f61eb9b897375de5323c5106128c8d6c"
Nov 02 09:45:18.093060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:18.092981    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c" event={"ID":"059ffd84-6945-4f53-a912-7fe1328708e3","Type":"ContainerStarted","Data":"3e3e504ba45b9881444de0a3a827553ae5e91e388ccb25c5b2b2f6f73f9969b3"}
Nov 02 09:45:18.113817 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:18.113774    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg" podStartSLOduration=18.113761217 podStartE2EDuration="18.113761217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:18.109587365 +0000 UTC m=+595.047874142" watchObservedRunningTime="2026-11-02 09:45:18.113761217 +0000 UTC m=+595.052048046"
Nov 02 09:45:18.180345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:18.180306    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"]
Nov 02 09:45:18.185643 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:18.185608    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e80fb-29893545-p9xp2"]
Nov 02 09:45:18.289349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:18.289304    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 09:45:18.289498 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:18.289372    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 09:45:19.105514 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:19.105477    4449 generic.go:356] "Generic (PLEG): container finished" podID="19995443-0610-41df-b899-dbc0ac2c0360" containerID="1a45676617fd059f462589534ac18a436a7de32919ceb4115d08bf18a798e1ff" exitCode=60
Nov 02 09:45:19.105887 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:19.105578    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg" event={"ID":"19995443-0610-41df-b899-dbc0ac2c0360","Type":"ContainerDied","Data":"1a45676617fd059f462589534ac18a436a7de32919ceb4115d08bf18a798e1ff"}
Nov 02 09:45:19.105887 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:19.105635    4449 scope.go:122] "RemoveContainer" containerID="5e4bd172169dde10e8d91a630c3d8aa122a5712bcd09ebf259c4a36f280d5d49"
Nov 02 09:45:19.106046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:19.106029    4449 scope.go:122] "RemoveContainer" containerID="1a45676617fd059f462589534ac18a436a7de32919ceb4115d08bf18a798e1ff"
Nov 02 09:45:19.106395 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:45:19.106369    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg_odh-ai-gateway-infra(19995443-0610-41df-b899-dbc0ac2c0360)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg" podUID="19995443-0610-41df-b899-dbc0ac2c0360"
Nov 02 09:45:19.111011 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:19.110983    4449 generic.go:356] "Generic (PLEG): container finished" podID="059ffd84-6945-4f53-a912-7fe1328708e3" containerID="3e3e504ba45b9881444de0a3a827553ae5e91e388ccb25c5b2b2f6f73f9969b3" exitCode=60
Nov 02 09:45:19.111142 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:19.111068    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c" event={"ID":"059ffd84-6945-4f53-a912-7fe1328708e3","Type":"ContainerDied","Data":"3e3e504ba45b9881444de0a3a827553ae5e91e388ccb25c5b2b2f6f73f9969b3"}
Nov 02 09:45:19.111709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:19.111690    4449 scope.go:122] "RemoveContainer" containerID="3e3e504ba45b9881444de0a3a827553ae5e91e388ccb25c5b2b2f6f73f9969b3"
Nov 02 09:45:19.112146 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:45:19.112122    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c_odh-ai-gateway-infra(059ffd84-6945-4f53-a912-7fe1328708e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c" podUID="059ffd84-6945-4f53-a912-7fe1328708e3"
Nov 02 09:45:19.126266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:19.125548    4449 scope.go:122] "RemoveContainer" containerID="f5819896937a4e1fe9ac2e205ad8a8ab0d54a37d54d660b755030e5400b79047"
Nov 02 09:45:19.130020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:19.127762    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"]
Nov 02 09:45:19.180403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:19.180317    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"]
Nov 02 09:45:19.209401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:19.209365    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ababc0-9768-4b79-a3e3-0c6f510aa287" path="/var/lib/kubelet/pods/f2ababc0-9768-4b79-a3e3-0c6f510aa287/volumes"
Nov 02 09:45:19.292150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:19.291653    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 09:45:19.292150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:19.291730    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 09:45:20.300435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.288670    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 09:45:20.300435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.288750    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 09:45:20.365518 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.365491    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"
Nov 02 09:45:20.416326 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.416288    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"
Nov 02 09:45:20.440350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.440307    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19995443-0610-41df-b899-dbc0ac2c0360-kube-api-access-6b5ft\" (UniqueName: \"kubernetes.io/projected/19995443-0610-41df-b899-dbc0ac2c0360-kube-api-access-6b5ft\") pod \"19995443-0610-41df-b899-dbc0ac2c0360\" (UID: \"19995443-0610-41df-b899-dbc0ac2c0360\") "
Nov 02 09:45:20.440553 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.440476    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19995443-0610-41df-b899-dbc0ac2c0360-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19995443-0610-41df-b899-dbc0ac2c0360-maas-api-service-ca\") pod \"19995443-0610-41df-b899-dbc0ac2c0360\" (UID: \"19995443-0610-41df-b899-dbc0ac2c0360\") "
Nov 02 09:45:20.441127 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.441074    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19995443-0610-41df-b899-dbc0ac2c0360-maas-api-service-ca" pod "19995443-0610-41df-b899-dbc0ac2c0360" (UID: "19995443-0610-41df-b899-dbc0ac2c0360"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:45:20.443471 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.443438    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19995443-0610-41df-b899-dbc0ac2c0360-kube-api-access-6b5ft" pod "19995443-0610-41df-b899-dbc0ac2c0360" (UID: "19995443-0610-41df-b899-dbc0ac2c0360"). InnerVolumeSpecName "kube-api-access-6b5ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:45:20.541874 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.541842    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059ffd84-6945-4f53-a912-7fe1328708e3-kube-api-access-fgp28\" (UniqueName: \"kubernetes.io/projected/059ffd84-6945-4f53-a912-7fe1328708e3-kube-api-access-fgp28\") pod \"059ffd84-6945-4f53-a912-7fe1328708e3\" (UID: \"059ffd84-6945-4f53-a912-7fe1328708e3\") "
Nov 02 09:45:20.542058 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.542032    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059ffd84-6945-4f53-a912-7fe1328708e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059ffd84-6945-4f53-a912-7fe1328708e3-maas-api-service-ca\") pod \"059ffd84-6945-4f53-a912-7fe1328708e3\" (UID: \"059ffd84-6945-4f53-a912-7fe1328708e3\") "
Nov 02 09:45:20.542329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.542312    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19995443-0610-41df-b899-dbc0ac2c0360-maas-api-service-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:20.542437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.542332    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b5ft\" (UniqueName: \"kubernetes.io/projected/19995443-0610-41df-b899-dbc0ac2c0360-kube-api-access-6b5ft\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:20.542437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.542416    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059ffd84-6945-4f53-a912-7fe1328708e3-maas-api-service-ca" pod "059ffd84-6945-4f53-a912-7fe1328708e3" (UID: "059ffd84-6945-4f53-a912-7fe1328708e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:45:20.544301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.544263    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059ffd84-6945-4f53-a912-7fe1328708e3-kube-api-access-fgp28" pod "059ffd84-6945-4f53-a912-7fe1328708e3" (UID: "059ffd84-6945-4f53-a912-7fe1328708e3"). InnerVolumeSpecName "kube-api-access-fgp28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:45:20.643813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.643713    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059ffd84-6945-4f53-a912-7fe1328708e3-maas-api-service-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:20.643813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:20.643757    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgp28\" (UniqueName: \"kubernetes.io/projected/059ffd84-6945-4f53-a912-7fe1328708e3-kube-api-access-fgp28\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:21.140597 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.140566    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"
Nov 02 09:45:21.140800 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.140610    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg" event={"ID":"19995443-0610-41df-b899-dbc0ac2c0360","Type":"ContainerDied","Data":"50bf9034a2b5b98586d686388d9527054ac5f23d144806e9c93b5e9d35b1e711"}
Nov 02 09:45:21.140800 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.140676    4449 scope.go:122] "RemoveContainer" containerID="1a45676617fd059f462589534ac18a436a7de32919ceb4115d08bf18a798e1ff"
Nov 02 09:45:21.142186 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.142167    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"
Nov 02 09:45:21.142330 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.142189    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c" event={"ID":"059ffd84-6945-4f53-a912-7fe1328708e3","Type":"ContainerDied","Data":"5ef66ea43598a81337b3fe7ee73171374f4bbf57d3ec74da3219d31663ea0c45"}
Nov 02 09:45:21.152149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.152116    4449 scope.go:122] "RemoveContainer" containerID="3e3e504ba45b9881444de0a3a827553ae5e91e388ccb25c5b2b2f6f73f9969b3"
Nov 02 09:45:21.179577 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.179544    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"]
Nov 02 09:45:21.188990 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.188965    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-bde944-29893545-bp6dg"]
Nov 02 09:45:21.199376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.199353    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"]
Nov 02 09:45:21.207191 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.207165    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19995443-0610-41df-b899-dbc0ac2c0360" path="/var/lib/kubelet/pods/19995443-0610-41df-b899-dbc0ac2c0360/volumes"
Nov 02 09:45:21.208104 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.208085    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-d2e277-29893545-mng8c"]
Nov 02 09:45:21.293741 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.293715    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:45:21.294073 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.293990    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:45:21.294771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:21.294752    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:45:23.209200 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:23.209171    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059ffd84-6945-4f53-a912-7fe1328708e3" path="/var/lib/kubelet/pods/059ffd84-6945-4f53-a912-7fe1328708e3/volumes"
Nov 02 09:45:25.120934 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.120897    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"]
Nov 02 09:45:25.121846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.121823    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f2ababc0-9768-4b79-a3e3-0c6f510aa287" containerName="cleanup"
Nov 02 09:45:25.121895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.121850    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ababc0-9768-4b79-a3e3-0c6f510aa287" containerName="cleanup"
Nov 02 09:45:25.121895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.121870    4449 cpu_manager.go:414] "containerMap: removing container" podUID="19995443-0610-41df-b899-dbc0ac2c0360" containerName="cleanup"
Nov 02 09:45:25.121895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.121877    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19995443-0610-41df-b899-dbc0ac2c0360" containerName="cleanup"
Nov 02 09:45:25.121996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.121911    4449 cpu_manager.go:414] "containerMap: removing container" podUID="059ffd84-6945-4f53-a912-7fe1328708e3" containerName="cleanup"
Nov 02 09:45:25.121996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.121919    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059ffd84-6945-4f53-a912-7fe1328708e3" containerName="cleanup"
Nov 02 09:45:25.121996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.121934    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f2ababc0-9768-4b79-a3e3-0c6f510aa287" containerName="cleanup"
Nov 02 09:45:25.121996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.121941    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ababc0-9768-4b79-a3e3-0c6f510aa287" containerName="cleanup"
Nov 02 09:45:25.121996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.121955    4449 cpu_manager.go:414] "containerMap: removing container" podUID="059ffd84-6945-4f53-a912-7fe1328708e3" containerName="cleanup"
Nov 02 09:45:25.121996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.121963    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059ffd84-6945-4f53-a912-7fe1328708e3" containerName="cleanup"
Nov 02 09:45:25.121996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.121977    4449 cpu_manager.go:414] "containerMap: removing container" podUID="19995443-0610-41df-b899-dbc0ac2c0360" containerName="cleanup"
Nov 02 09:45:25.121996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.121984    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19995443-0610-41df-b899-dbc0ac2c0360" containerName="cleanup"
Nov 02 09:45:25.122309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.122004    4449 cpu_manager.go:414] "containerMap: removing container" podUID="059ffd84-6945-4f53-a912-7fe1328708e3" containerName="cleanup"
Nov 02 09:45:25.122309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.122010    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059ffd84-6945-4f53-a912-7fe1328708e3" containerName="cleanup"
Nov 02 09:45:25.122309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.122136    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="059ffd84-6945-4f53-a912-7fe1328708e3" containerName="cleanup"
Nov 02 09:45:25.122309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.122150    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="19995443-0610-41df-b899-dbc0ac2c0360" containerName="cleanup"
Nov 02 09:45:25.122309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.122160    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ababc0-9768-4b79-a3e3-0c6f510aa287" containerName="cleanup"
Nov 02 09:45:25.122309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.122168    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="059ffd84-6945-4f53-a912-7fe1328708e3" containerName="cleanup"
Nov 02 09:45:25.122309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.122178    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="19995443-0610-41df-b899-dbc0ac2c0360" containerName="cleanup"
Nov 02 09:45:25.122309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.122189    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ababc0-9768-4b79-a3e3-0c6f510aa287" containerName="cleanup"
Nov 02 09:45:25.122309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.122200    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="19995443-0610-41df-b899-dbc0ac2c0360" containerName="cleanup"
Nov 02 09:45:25.127551 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.127532    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.131988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.131965    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"]
Nov 02 09:45:25.133858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.133838    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"kube-root-ca.crt\""
Nov 02 09:45:25.133858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.133840    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"e2e-conflict-a-3b52f1e5-kserve-self-signed-certs\""
Nov 02 09:45:25.133858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.133857    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"openshift-service-ca.crt\""
Nov 02 09:45:25.134046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.133921    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"default-dockercfg-xf5qm\""
Nov 02 09:45:25.192865 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.192835    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m8h9\" (UniqueName: \"kubernetes.io/projected/be9f6a5d-188a-420e-aadc-d1deec2bf014-kube-api-access-7m8h9\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.193024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.192883    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-dshm\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.193024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.193001    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-home\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.193104 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.193041    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-model-cache\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.193104 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.193087    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-tmp-dir\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.193168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.193105    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be9f6a5d-188a-420e-aadc-d1deec2bf014-tls-certs\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.294032 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.293993    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m8h9\" (UniqueName: \"kubernetes.io/projected/be9f6a5d-188a-420e-aadc-d1deec2bf014-kube-api-access-7m8h9\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.294275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.294052    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-dshm\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.294275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.294110    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-home\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.294275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.294154    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-model-cache\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.294275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.294185    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-tmp-dir\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.294275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.294228    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be9f6a5d-188a-420e-aadc-d1deec2bf014-tls-certs\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.294704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.294680    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-model-cache\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.294773 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.294720    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-home\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.294809 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.294788    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-tmp-dir\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.296627 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.296608    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-dshm\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.297077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.297060    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be9f6a5d-188a-420e-aadc-d1deec2bf014-tls-certs\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.307905 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.307874    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m8h9\" (UniqueName: \"kubernetes.io/projected/be9f6a5d-188a-420e-aadc-d1deec2bf014-kube-api-access-7m8h9\") pod \"e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.441572 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.441484    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:25.748027 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:25.747994    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"]
Nov 02 09:45:25.748793 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:45:25.748760    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe9f6a5d_188a_420e_aadc_d1deec2bf014.slice/crio-bb07dfc2a5a0f400f9e36933c73e78d5303b8ee56114174e91810495613b1061 WatchSource:0}: Error finding container bb07dfc2a5a0f400f9e36933c73e78d5303b8ee56114174e91810495613b1061: Status 404 returned error can't find the container with id bb07dfc2a5a0f400f9e36933c73e78d5303b8ee56114174e91810495613b1061
Nov 02 09:45:26.181047 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:26.181011    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q" event={"ID":"be9f6a5d-188a-420e-aadc-d1deec2bf014","Type":"ContainerStarted","Data":"9a7c58ea0311640eea64f359f28fd13a34c7b9d83bf13f1fc467a80bcbec2740"}
Nov 02 09:45:26.181472 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:26.181058    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q" event={"ID":"be9f6a5d-188a-420e-aadc-d1deec2bf014","Type":"ContainerStarted","Data":"bb07dfc2a5a0f400f9e36933c73e78d5303b8ee56114174e91810495613b1061"}
Nov 02 09:45:26.205916 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:26.205867    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q" podStartSLOduration=1.205854643 podStartE2EDuration="1.205854643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:26.203918192 +0000 UTC m=+603.142204997" watchObservedRunningTime="2026-11-02 09:45:26.205854643 +0000 UTC m=+603.144141463"
Nov 02 09:45:26.442208 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:26.442120    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:28.097006 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:28.096966    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"
Nov 02 09:45:29.120920 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:29.120561    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"
Nov 02 09:45:35.277037 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:35.277004    4449 generic.go:356] "Generic (PLEG): container finished" podID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerID="fc47298d4dbb14070fd07471ea1613b082cf73f8b47a7ced1dbf45924739a10d" exitCode=0
Nov 02 09:45:35.277622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:35.277070    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" event={"ID":"0c312869-83d6-406b-bff4-6dc3068df5bf","Type":"ContainerDied","Data":"fc47298d4dbb14070fd07471ea1613b082cf73f8b47a7ced1dbf45924739a10d"}
Nov 02 09:45:35.277622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:35.277490    4449 scope.go:122] "RemoveContainer" containerID="fc47298d4dbb14070fd07471ea1613b082cf73f8b47a7ced1dbf45924739a10d"
Nov 02 09:45:35.278934 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:35.278904    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:45:35.279004 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:35.278977    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:45:35.279039 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:35.279011    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:45:35.442106 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:35.442074    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:35.455159 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:35.455132    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:36.316342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:36.312626    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" event={"ID":"0c312869-83d6-406b-bff4-6dc3068df5bf","Type":"ContainerStarted","Data":"2a00e9bcc65ad9494ec7b859ae35cd64b37b4985c196c0e9f6857d02d9fd5e86"}
Nov 02 09:45:36.329776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:36.329749    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:45:36.347021 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:36.347001    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:36.347263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:36.347205    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 09:45:36.347363 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:36.347292    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 09:45:36.546739 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:45:36.546702    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:45:37.347411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:37.347380    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 09:45:37.347799 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:37.347427    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 09:45:38.347939 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:38.347905    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 09:45:38.348351 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:38.347962    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 09:45:39.347908 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:39.347872    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 09:45:39.348072 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:39.347952    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 09:45:40.347364 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:40.347329    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 09:45:40.347544 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:40.347388    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 09:45:41.348970 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:41.348817    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 09:45:41.348970 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:41.348879    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 09:45:42.352260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.352230    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:42.352703 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.352556    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:42.353416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.353394    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:45:42.776772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.776688    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"]
Nov 02 09:45:42.777277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.777256    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f2ababc0-9768-4b79-a3e3-0c6f510aa287" containerName="cleanup"
Nov 02 09:45:42.777277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.777275    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ababc0-9768-4b79-a3e3-0c6f510aa287" containerName="cleanup"
Nov 02 09:45:42.777456 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.777314    4449 cpu_manager.go:414] "containerMap: removing container" podUID="19995443-0610-41df-b899-dbc0ac2c0360" containerName="cleanup"
Nov 02 09:45:42.777456 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.777322    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19995443-0610-41df-b899-dbc0ac2c0360" containerName="cleanup"
Nov 02 09:45:42.777456 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.777406    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="059ffd84-6945-4f53-a912-7fe1328708e3" containerName="cleanup"
Nov 02 09:45:42.777456 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.777417    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ababc0-9768-4b79-a3e3-0c6f510aa287" containerName="cleanup"
Nov 02 09:45:42.780555 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.780538    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.789957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.789934    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"e2e-conflict-b-3b52f1e5-kserve-self-signed-certs\""
Nov 02 09:45:42.791098 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.791077    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"]
Nov 02 09:45:42.879535 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.879508    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-model-cache\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.879692 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.879540    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-home\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.879692 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.879579    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tls-certs\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.879692 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.879668    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tmp-dir\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.879692 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.879688    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7tc\" (UniqueName: \"kubernetes.io/projected/f1575768-6b2c-4026-bfd2-5aaa838f36a5-kube-api-access-8c7tc\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.879826 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.879712    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-dshm\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.980606 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.980576    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-model-cache\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.980606 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.980612    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-home\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.980842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.980643    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tls-certs\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.980842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.980692    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tmp-dir\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.980842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.980711    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7tc\" (UniqueName: \"kubernetes.io/projected/f1575768-6b2c-4026-bfd2-5aaa838f36a5-kube-api-access-8c7tc\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.980842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.980743    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-dshm\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.981085 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.981061    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-model-cache\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.981153 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.981106    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-home\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.981153 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.981127    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tmp-dir\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.983192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.983166    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-dshm\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.983369 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.983352    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tls-certs\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:42.993883 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:42.993862    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7tc\" (UniqueName: \"kubernetes.io/projected/f1575768-6b2c-4026-bfd2-5aaa838f36a5-kube-api-access-8c7tc\") pod \"e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") " pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:43.093797 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:43.093750    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:43.372564 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:43.372538    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"]
Nov 02 09:45:43.374140 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:45:43.374113    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1575768_6b2c_4026_bfd2_5aaa838f36a5.slice/crio-298199ec2435e1c7400b52bacee1e3b80afc2f7d0e0a05446db4677ca1e55651 WatchSource:0}: Error finding container 298199ec2435e1c7400b52bacee1e3b80afc2f7d0e0a05446db4677ca1e55651: Status 404 returned error can't find the container with id 298199ec2435e1c7400b52bacee1e3b80afc2f7d0e0a05446db4677ca1e55651
Nov 02 09:45:44.378263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:44.378209    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652" event={"ID":"f1575768-6b2c-4026-bfd2-5aaa838f36a5","Type":"ContainerStarted","Data":"fad8eb2bdd90dc7e734e22f1287b836df6acc9d4cbb24eb2ac20b2d8253166a6"}
Nov 02 09:45:44.378714 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:44.378274    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652" event={"ID":"f1575768-6b2c-4026-bfd2-5aaa838f36a5","Type":"ContainerStarted","Data":"298199ec2435e1c7400b52bacee1e3b80afc2f7d0e0a05446db4677ca1e55651"}
Nov 02 09:45:44.398294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:44.398243    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652" podStartSLOduration=2.39820445 podStartE2EDuration="2.39820445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:44.397620725 +0000 UTC m=+621.335907527" watchObservedRunningTime="2026-11-02 09:45:44.39820445 +0000 UTC m=+621.336491220"
Nov 02 09:45:45.094889 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:45.094857    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:53.094870 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.094834    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:53.107233 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.107190    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:53.253105 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.253067    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"]
Nov 02 09:45:53.258527 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.258506    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.265968 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.265733    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-000ffb-openshift-default-dockercfg-hz5bc\""
Nov 02 09:45:53.270204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.270180    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"]
Nov 02 09:45:53.403297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.403174    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe7331b-6724-4406-82bc-814e9dc8179c-istiod-ca-cert\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.403297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.403249    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-podinfo\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.403482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.403348    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-certs\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.403482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.403385    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-envoy\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.403482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.403456    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvhg2\" (UniqueName: \"kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-kube-api-access-rvhg2\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.403578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.403552    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-token\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.403609 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.403582    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-credential-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.403644 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.403616    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.403677 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.403651    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-data\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.453552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.453518    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:45:53.504399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.504357    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-certs\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.504573 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.504407    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-envoy\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.504573 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.504434    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvhg2\" (UniqueName: \"kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-kube-api-access-rvhg2\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.504573 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.504547    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-token\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.504768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.504576    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-credential-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.504768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.504600    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.504768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.504627    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-data\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.504768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.504695    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe7331b-6724-4406-82bc-814e9dc8179c-istiod-ca-cert\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.504768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.504720    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-podinfo\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.505029 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.504817    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-certs\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.505304 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.505274    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-credential-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.505593 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.505348    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-data\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.505681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.505587    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.506289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.506266    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe7331b-6724-4406-82bc-814e9dc8179c-istiod-ca-cert\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.507760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.507734    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-podinfo\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.508227 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.508187    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-envoy\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.517790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.517764    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-token\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.518005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.517984    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvhg2\" (UniqueName: \"kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-kube-api-access-rvhg2\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.574176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.574146    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:53.874540 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.874514    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"]
Nov 02 09:45:53.875643 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:45:53.875614    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe7331b_6724_4406_82bc_814e9dc8179c.slice/crio-81f82ce3926f6436c433e0cd9756230f624a0681d52c77743959a367300343b7 WatchSource:0}: Error finding container 81f82ce3926f6436c433e0cd9756230f624a0681d52c77743959a367300343b7: Status 404 returned error can't find the container with id 81f82ce3926f6436c433e0cd9756230f624a0681d52c77743959a367300343b7
Nov 02 09:45:53.878344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.878307    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:45:53.878419 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.878391    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:45:53.878505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:53.878431    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:45:54.456817 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:54.456780    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw" event={"ID":"1fe7331b-6724-4406-82bc-814e9dc8179c","Type":"ContainerStarted","Data":"5e12f7b82a63d2d1dd12fbae2b20650fae5634af4d86024cb26bbea4e495ec72"}
Nov 02 09:45:54.457449 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:54.457423    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw" event={"ID":"1fe7331b-6724-4406-82bc-814e9dc8179c","Type":"ContainerStarted","Data":"81f82ce3926f6436c433e0cd9756230f624a0681d52c77743959a367300343b7"}
Nov 02 09:45:54.485925 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:54.485860    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw" podStartSLOduration=1.485844978 podStartE2EDuration="1.485844978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:54.481257908 +0000 UTC m=+631.419544711" watchObservedRunningTime="2026-11-02 09:45:54.485844978 +0000 UTC m=+631.424131753"
Nov 02 09:45:54.574748 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:54.574617    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:45:54.576382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:54.576344    4449 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 09:45:54.576569 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:54.576421    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw" podUID="1fe7331b-6724-4406-82bc-814e9dc8179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 09:45:55.575241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:55.574677    4449 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 09:45:55.575241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:55.574761    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw" podUID="1fe7331b-6724-4406-82bc-814e9dc8179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 09:45:56.574689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:56.574657    4449 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 09:45:56.574904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:56.574709    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw" podUID="1fe7331b-6724-4406-82bc-814e9dc8179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 09:45:57.574687 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:57.574643    4449 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 09:45:57.575102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:57.574708    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw" podUID="1fe7331b-6724-4406-82bc-814e9dc8179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 09:45:58.574642 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:58.574603    4449 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 09:45:58.574848 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:58.574712    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw" podUID="1fe7331b-6724-4406-82bc-814e9dc8179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 09:45:59.574646 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:59.574606    4449 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 09:45:59.574911 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:45:59.574885    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw" podUID="1fe7331b-6724-4406-82bc-814e9dc8179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 09:46:00.580335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:00.580305    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:46:00.580920 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:00.580794    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:46:00.581699 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:00.581679    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:46:03.169722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:03.169689    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"]
Nov 02 09:46:06.546901 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:46:06.546864    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:46:07.417900 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:07.417854    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"]
Nov 02 09:46:18.190967 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.190924    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652" podUID="f1575768-6b2c-4026-bfd2-5aaa838f36a5" containerName="main" containerID="cri-o://fad8eb2bdd90dc7e734e22f1287b836df6acc9d4cbb24eb2ac20b2d8253166a6" gracePeriod=45
Nov 02 09:46:18.461892 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.461867    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:46:18.539151 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.539114    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tls-certs\") pod \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") "
Nov 02 09:46:18.539387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.539166    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-model-cache\") pod \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") "
Nov 02 09:46:18.539387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.539196    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-dshm\") pod \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") "
Nov 02 09:46:18.539387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.539296    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tmp-dir\") pod \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") "
Nov 02 09:46:18.539564 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.539407    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1575768-6b2c-4026-bfd2-5aaa838f36a5-kube-api-access-8c7tc\" (UniqueName: \"kubernetes.io/projected/f1575768-6b2c-4026-bfd2-5aaa838f36a5-kube-api-access-8c7tc\") pod \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") "
Nov 02 09:46:18.539564 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.539480    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-home\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-home\") pod \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\" (UID: \"f1575768-6b2c-4026-bfd2-5aaa838f36a5\") "
Nov 02 09:46:18.539564 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.539529    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-model-cache" pod "f1575768-6b2c-4026-bfd2-5aaa838f36a5" (UID: "f1575768-6b2c-4026-bfd2-5aaa838f36a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:18.539701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.539669    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tmp-dir" pod "f1575768-6b2c-4026-bfd2-5aaa838f36a5" (UID: "f1575768-6b2c-4026-bfd2-5aaa838f36a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:18.539784 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.539760    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-home" pod "f1575768-6b2c-4026-bfd2-5aaa838f36a5" (UID: "f1575768-6b2c-4026-bfd2-5aaa838f36a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:18.540012 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.539992    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:18.540012 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.540010    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:18.540397 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.540022    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:18.541761 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.541734    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-dshm" pod "f1575768-6b2c-4026-bfd2-5aaa838f36a5" (UID: "f1575768-6b2c-4026-bfd2-5aaa838f36a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:18.541852 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.541792    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1575768-6b2c-4026-bfd2-5aaa838f36a5-kube-api-access-8c7tc" pod "f1575768-6b2c-4026-bfd2-5aaa838f36a5" (UID: "f1575768-6b2c-4026-bfd2-5aaa838f36a5"). InnerVolumeSpecName "kube-api-access-8c7tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:18.541852 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.541803    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tls-certs" pod "f1575768-6b2c-4026-bfd2-5aaa838f36a5" (UID: "f1575768-6b2c-4026-bfd2-5aaa838f36a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:46:18.641013 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.640976    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c7tc\" (UniqueName: \"kubernetes.io/projected/f1575768-6b2c-4026-bfd2-5aaa838f36a5-kube-api-access-8c7tc\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:18.641013 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.641009    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1575768-6b2c-4026-bfd2-5aaa838f36a5-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:18.641013 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.641019    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1575768-6b2c-4026-bfd2-5aaa838f36a5-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:18.660403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.660364    4449 generic.go:356] "Generic (PLEG): container finished" podID="f1575768-6b2c-4026-bfd2-5aaa838f36a5" containerID="fad8eb2bdd90dc7e734e22f1287b836df6acc9d4cbb24eb2ac20b2d8253166a6" exitCode=0
Nov 02 09:46:18.660580 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.660414    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"
Nov 02 09:46:18.660580 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.660420    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652" event={"ID":"f1575768-6b2c-4026-bfd2-5aaa838f36a5","Type":"ContainerDied","Data":"fad8eb2bdd90dc7e734e22f1287b836df6acc9d4cbb24eb2ac20b2d8253166a6"}
Nov 02 09:46:18.660580 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.660467    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652" event={"ID":"f1575768-6b2c-4026-bfd2-5aaa838f36a5","Type":"ContainerDied","Data":"298199ec2435e1c7400b52bacee1e3b80afc2f7d0e0a05446db4677ca1e55651"}
Nov 02 09:46:18.660580 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.660490    4449 scope.go:122] "RemoveContainer" containerID="fad8eb2bdd90dc7e734e22f1287b836df6acc9d4cbb24eb2ac20b2d8253166a6"
Nov 02 09:46:18.679464 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.679433    4449 scope.go:122] "RemoveContainer" containerID="fad8eb2bdd90dc7e734e22f1287b836df6acc9d4cbb24eb2ac20b2d8253166a6"
Nov 02 09:46:18.679818 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:46:18.679796    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad8eb2bdd90dc7e734e22f1287b836df6acc9d4cbb24eb2ac20b2d8253166a6\": container with ID starting with fad8eb2bdd90dc7e734e22f1287b836df6acc9d4cbb24eb2ac20b2d8253166a6 not found: ID does not exist" containerID="fad8eb2bdd90dc7e734e22f1287b836df6acc9d4cbb24eb2ac20b2d8253166a6"
Nov 02 09:46:18.679898 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.679832    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad8eb2bdd90dc7e734e22f1287b836df6acc9d4cbb24eb2ac20b2d8253166a6"} err="failed to get container status \"fad8eb2bdd90dc7e734e22f1287b836df6acc9d4cbb24eb2ac20b2d8253166a6\": rpc error: code = NotFound desc = could not find container \"fad8eb2bdd90dc7e734e22f1287b836df6acc9d4cbb24eb2ac20b2d8253166a6\": container with ID starting with fad8eb2bdd90dc7e734e22f1287b836df6acc9d4cbb24eb2ac20b2d8253166a6 not found: ID does not exist"
Nov 02 09:46:18.691140 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.691102    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"]
Nov 02 09:46:18.697610 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:18.697532    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652"]
Nov 02 09:46:19.206568 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:19.206530    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1575768-6b2c-4026-bfd2-5aaa838f36a5" path="/var/lib/kubelet/pods/f1575768-6b2c-4026-bfd2-5aaa838f36a5/volumes"
Nov 02 09:46:19.443639 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:19.443595    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-b-3b52f1e5-kserve-5c5848bbd6-qw652" podUID="f1575768-6b2c-4026-bfd2-5aaa838f36a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:46:20.046746 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:20.046704    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7360-0dc00d"]
Nov 02 09:46:20.050462 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:20.047942    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f1575768-6b2c-4026-bfd2-5aaa838f36a5" containerName="main"
Nov 02 09:46:20.050462 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:20.047973    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1575768-6b2c-4026-bfd2-5aaa838f36a5" containerName="main"
Nov 02 09:46:20.050462 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:20.048110    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1575768-6b2c-4026-bfd2-5aaa838f36a5" containerName="main"
Nov 02 09:46:20.053478 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:20.053035    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7360-0dc00d"
Nov 02 09:46:20.057149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:20.057104    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrd62\" (UniqueName: \"kubernetes.io/projected/1149bbf4-0d14-4db3-b3ad-c43872be93e1-kube-api-access-nrd62\") pod \"test-curl-7360-0dc00d\" (UID: \"1149bbf4-0d14-4db3-b3ad-c43872be93e1\") " pod="opendatahub/test-curl-7360-0dc00d"
Nov 02 09:46:20.057329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:20.057253    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7360-0dc00d"]
Nov 02 09:46:20.058431 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:20.058399    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8bqk\""
Nov 02 09:46:20.158324 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:20.158286    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrd62\" (UniqueName: \"kubernetes.io/projected/1149bbf4-0d14-4db3-b3ad-c43872be93e1-kube-api-access-nrd62\") pod \"test-curl-7360-0dc00d\" (UID: \"1149bbf4-0d14-4db3-b3ad-c43872be93e1\") " pod="opendatahub/test-curl-7360-0dc00d"
Nov 02 09:46:20.177798 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:20.177764    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrd62\" (UniqueName: \"kubernetes.io/projected/1149bbf4-0d14-4db3-b3ad-c43872be93e1-kube-api-access-nrd62\") pod \"test-curl-7360-0dc00d\" (UID: \"1149bbf4-0d14-4db3-b3ad-c43872be93e1\") " pod="opendatahub/test-curl-7360-0dc00d"
Nov 02 09:46:20.370423 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:20.370329    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7360-0dc00d"
Nov 02 09:46:20.671995 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:20.671960    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7360-0dc00d"]
Nov 02 09:46:20.675095 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:46:20.674437    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1149bbf4_0d14_4db3_b3ad_c43872be93e1.slice/crio-2abed2e534b617daab69e01d35918cd1eb19c79fbd62f5971633fc8372f3c411 WatchSource:0}: Error finding container 2abed2e534b617daab69e01d35918cd1eb19c79fbd62f5971633fc8372f3c411: Status 404 returned error can't find the container with id 2abed2e534b617daab69e01d35918cd1eb19c79fbd62f5971633fc8372f3c411
Nov 02 09:46:21.694400 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:21.694357    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7360-0dc00d" event={"ID":"1149bbf4-0d14-4db3-b3ad-c43872be93e1","Type":"ContainerStarted","Data":"2abed2e534b617daab69e01d35918cd1eb19c79fbd62f5971633fc8372f3c411"}
Nov 02 09:46:22.328046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.327772    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"]
Nov 02 09:46:22.438898 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.438854    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q" podUID="be9f6a5d-188a-420e-aadc-d1deec2bf014" containerName="main" containerID="cri-o://9a7c58ea0311640eea64f359f28fd13a34c7b9d83bf13f1fc467a80bcbec2740" gracePeriod=45
Nov 02 09:46:22.748676 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.745670    4449 generic.go:356] "Generic (PLEG): container finished" podID="be9f6a5d-188a-420e-aadc-d1deec2bf014" containerID="9a7c58ea0311640eea64f359f28fd13a34c7b9d83bf13f1fc467a80bcbec2740" exitCode=0
Nov 02 09:46:22.748676 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.745829    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q" event={"ID":"be9f6a5d-188a-420e-aadc-d1deec2bf014","Type":"ContainerDied","Data":"9a7c58ea0311640eea64f359f28fd13a34c7b9d83bf13f1fc467a80bcbec2740"}
Nov 02 09:46:22.817627 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.816567    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:46:22.827162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.827116    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-model-cache\") pod \"be9f6a5d-188a-420e-aadc-d1deec2bf014\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") "
Nov 02 09:46:22.827361 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.827254    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-dshm\") pod \"be9f6a5d-188a-420e-aadc-d1deec2bf014\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") "
Nov 02 09:46:22.827434 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.827371    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be9f6a5d-188a-420e-aadc-d1deec2bf014-tls-certs\" (UniqueName: \"kubernetes.io/secret/be9f6a5d-188a-420e-aadc-d1deec2bf014-tls-certs\") pod \"be9f6a5d-188a-420e-aadc-d1deec2bf014\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") "
Nov 02 09:46:22.827472 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.827447    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-home\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-home\") pod \"be9f6a5d-188a-420e-aadc-d1deec2bf014\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") "
Nov 02 09:46:22.827507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.827490    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-tmp-dir\") pod \"be9f6a5d-188a-420e-aadc-d1deec2bf014\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") "
Nov 02 09:46:22.827595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.827578    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9f6a5d-188a-420e-aadc-d1deec2bf014-kube-api-access-7m8h9\" (UniqueName: \"kubernetes.io/projected/be9f6a5d-188a-420e-aadc-d1deec2bf014-kube-api-access-7m8h9\") pod \"be9f6a5d-188a-420e-aadc-d1deec2bf014\" (UID: \"be9f6a5d-188a-420e-aadc-d1deec2bf014\") "
Nov 02 09:46:22.829158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.829123    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-home" pod "be9f6a5d-188a-420e-aadc-d1deec2bf014" (UID: "be9f6a5d-188a-420e-aadc-d1deec2bf014"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:22.829495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.829462    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-model-cache" pod "be9f6a5d-188a-420e-aadc-d1deec2bf014" (UID: "be9f6a5d-188a-420e-aadc-d1deec2bf014"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:22.829579 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.829494    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-tmp-dir" pod "be9f6a5d-188a-420e-aadc-d1deec2bf014" (UID: "be9f6a5d-188a-420e-aadc-d1deec2bf014"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:22.835616 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.835584    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-dshm" pod "be9f6a5d-188a-420e-aadc-d1deec2bf014" (UID: "be9f6a5d-188a-420e-aadc-d1deec2bf014"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:22.836509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.836474    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9f6a5d-188a-420e-aadc-d1deec2bf014-kube-api-access-7m8h9" pod "be9f6a5d-188a-420e-aadc-d1deec2bf014" (UID: "be9f6a5d-188a-420e-aadc-d1deec2bf014"). InnerVolumeSpecName "kube-api-access-7m8h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:22.838683 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.838654    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9f6a5d-188a-420e-aadc-d1deec2bf014-tls-certs" pod "be9f6a5d-188a-420e-aadc-d1deec2bf014" (UID: "be9f6a5d-188a-420e-aadc-d1deec2bf014"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:46:22.929172 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.929075    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be9f6a5d-188a-420e-aadc-d1deec2bf014-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:22.929172 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.929107    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:22.929172 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.929140    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:22.929172 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.929150    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m8h9\" (UniqueName: \"kubernetes.io/projected/be9f6a5d-188a-420e-aadc-d1deec2bf014-kube-api-access-7m8h9\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:22.929172 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.929161    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:22.929172 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:22.929169    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be9f6a5d-188a-420e-aadc-d1deec2bf014-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:23.753435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:23.753402    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"
Nov 02 09:46:23.753435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:23.753414    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q" event={"ID":"be9f6a5d-188a-420e-aadc-d1deec2bf014","Type":"ContainerDied","Data":"bb07dfc2a5a0f400f9e36933c73e78d5303b8ee56114174e91810495613b1061"}
Nov 02 09:46:23.753988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:23.753475    4449 scope.go:122] "RemoveContainer" containerID="9a7c58ea0311640eea64f359f28fd13a34c7b9d83bf13f1fc467a80bcbec2740"
Nov 02 09:46:23.774694 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:23.774664    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"]
Nov 02 09:46:23.779195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:23.779171    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/e2e-conflict-a-3b52f1e5-kserve-7cc5496456-mc47q"]
Nov 02 09:46:24.762101 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:24.762060    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7360-0dc00d" event={"ID":"1149bbf4-0d14-4db3-b3ad-c43872be93e1","Type":"ContainerStarted","Data":"3dfb27ffb224cc1f7e93621775317c0b91bd3ba0cc660ed29a7879abcfa8eaa5"}
Nov 02 09:46:24.778899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:24.778850    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7360-0dc00d" podStartSLOduration=1.622668012 podStartE2EDuration="4.77883565s" totalImagesPullingTime="3.156167638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:24.778073928 +0000 UTC m=+661.716360707" watchObservedRunningTime="2026-11-02 09:46:24.77883565 +0000 UTC m=+661.717122425"
Nov 02 09:46:25.206969 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.206935    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9f6a5d-188a-420e-aadc-d1deec2bf014" path="/var/lib/kubelet/pods/be9f6a5d-188a-420e-aadc-d1deec2bf014/volumes"
Nov 02 09:46:25.511510 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.511420    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7360-0dc00d"]
Nov 02 09:46:25.516206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.516178    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7360-0dc00d"]
Nov 02 09:46:25.683785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.683752    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7360-90a43e"]
Nov 02 09:46:25.684356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.684340    4449 cpu_manager.go:414] "containerMap: removing container" podUID="be9f6a5d-188a-420e-aadc-d1deec2bf014" containerName="main"
Nov 02 09:46:25.684411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.684358    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9f6a5d-188a-420e-aadc-d1deec2bf014" containerName="main"
Nov 02 09:46:25.684411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.684376    4449 cpu_manager.go:414] "containerMap: removing container" podUID="1149bbf4-0d14-4db3-b3ad-c43872be93e1" containerName="test-curl-7360-0dc00d"
Nov 02 09:46:25.684411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.684382    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1149bbf4-0d14-4db3-b3ad-c43872be93e1" containerName="test-curl-7360-0dc00d"
Nov 02 09:46:25.684524 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.684490    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9f6a5d-188a-420e-aadc-d1deec2bf014" containerName="main"
Nov 02 09:46:25.684524 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.684499    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="1149bbf4-0d14-4db3-b3ad-c43872be93e1" containerName="test-curl-7360-0dc00d"
Nov 02 09:46:25.688858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.688837    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7360-90a43e"
Nov 02 09:46:25.698587 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.698561    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7360-90a43e"]
Nov 02 09:46:25.760322 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.760269    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wj8t\" (UniqueName: \"kubernetes.io/projected/86ec625d-41e3-40ea-94a2-bb18b3006d3a-kube-api-access-7wj8t\") pod \"test-curl-7360-90a43e\" (UID: \"86ec625d-41e3-40ea-94a2-bb18b3006d3a\") " pod="opendatahub/test-curl-7360-90a43e"
Nov 02 09:46:25.844705 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.844672    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"]
Nov 02 09:46:25.848823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.848801    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"
Nov 02 09:46:25.853125 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.853097    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:46:25.853333 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.853137    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:46:25.853333 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.853141    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:46:25.853519 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.853173    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:46:25.856604 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.856578    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"]
Nov 02 09:46:25.861615 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.861589    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wj8t\" (UniqueName: \"kubernetes.io/projected/86ec625d-41e3-40ea-94a2-bb18b3006d3a-kube-api-access-7wj8t\") pod \"test-curl-7360-90a43e\" (UID: \"86ec625d-41e3-40ea-94a2-bb18b3006d3a\") " pod="opendatahub/test-curl-7360-90a43e"
Nov 02 09:46:25.874780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.874747    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wj8t\" (UniqueName: \"kubernetes.io/projected/86ec625d-41e3-40ea-94a2-bb18b3006d3a-kube-api-access-7wj8t\") pod \"test-curl-7360-90a43e\" (UID: \"86ec625d-41e3-40ea-94a2-bb18b3006d3a\") " pod="opendatahub/test-curl-7360-90a43e"
Nov 02 09:46:25.963158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.963112    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncqkz\" (UniqueName: \"kubernetes.io/projected/af5227c4-b584-4416-86c2-8a94d61db2cd-kube-api-access-ncqkz\") pod \"maas-api-revoke-sub-eb8bfdd0-vrr4j\" (UID: \"af5227c4-b584-4416-86c2-8a94d61db2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"
Nov 02 09:46:25.963376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.963288    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8bfdd0-vrr4j\" (UID: \"af5227c4-b584-4416-86c2-8a94d61db2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"
Nov 02 09:46:25.963376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:25.963328    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8bfdd0-vrr4j\" (UID: \"af5227c4-b584-4416-86c2-8a94d61db2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"
Nov 02 09:46:26.004350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.004316    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7360-90a43e"
Nov 02 09:46:26.065146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.065105    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8bfdd0-vrr4j\" (UID: \"af5227c4-b584-4416-86c2-8a94d61db2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"
Nov 02 09:46:26.065317 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.065175    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8bfdd0-vrr4j\" (UID: \"af5227c4-b584-4416-86c2-8a94d61db2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"
Nov 02 09:46:26.067122 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.067095    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncqkz\" (UniqueName: \"kubernetes.io/projected/af5227c4-b584-4416-86c2-8a94d61db2cd-kube-api-access-ncqkz\") pod \"maas-api-revoke-sub-eb8bfdd0-vrr4j\" (UID: \"af5227c4-b584-4416-86c2-8a94d61db2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"
Nov 02 09:46:26.067761 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.067734    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8bfdd0-vrr4j\" (UID: \"af5227c4-b584-4416-86c2-8a94d61db2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"
Nov 02 09:46:26.078063 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.069943    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8bfdd0-vrr4j\" (UID: \"af5227c4-b584-4416-86c2-8a94d61db2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"
Nov 02 09:46:26.086137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.085407    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncqkz\" (UniqueName: \"kubernetes.io/projected/af5227c4-b584-4416-86c2-8a94d61db2cd-kube-api-access-ncqkz\") pod \"maas-api-revoke-sub-eb8bfdd0-vrr4j\" (UID: \"af5227c4-b584-4416-86c2-8a94d61db2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"
Nov 02 09:46:26.177302 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.169311    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"
Nov 02 09:46:26.298918 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:46:26.298790    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86ec625d_41e3_40ea_94a2_bb18b3006d3a.slice/crio-a911ebeb9443f780a3c73ff53c27b1ffe6b7969c752ecdda1268abb6678bffb1 WatchSource:0}: Error finding container a911ebeb9443f780a3c73ff53c27b1ffe6b7969c752ecdda1268abb6678bffb1: Status 404 returned error can't find the container with id a911ebeb9443f780a3c73ff53c27b1ffe6b7969c752ecdda1268abb6678bffb1
Nov 02 09:46:26.299404 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.299381    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7360-90a43e"]
Nov 02 09:46:26.505350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.505322    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"]
Nov 02 09:46:26.506380 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:46:26.506350    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5227c4_b584_4416_86c2_8a94d61db2cd.slice/crio-f7a58574e59d0b254f8c801ae452d89a30c06f1ccd168372ab6fe3ae371987f1 WatchSource:0}: Error finding container f7a58574e59d0b254f8c801ae452d89a30c06f1ccd168372ab6fe3ae371987f1: Status 404 returned error can't find the container with id f7a58574e59d0b254f8c801ae452d89a30c06f1ccd168372ab6fe3ae371987f1
Nov 02 09:46:26.777324 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.777288    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7360-90a43e" event={"ID":"86ec625d-41e3-40ea-94a2-bb18b3006d3a","Type":"ContainerStarted","Data":"d3b8b21a6c9a23aefb503544d033a816d37dbfd80136f3f47b6acf4e54c6dfaa"}
Nov 02 09:46:26.777488 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.777334    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7360-90a43e" event={"ID":"86ec625d-41e3-40ea-94a2-bb18b3006d3a","Type":"ContainerStarted","Data":"a911ebeb9443f780a3c73ff53c27b1ffe6b7969c752ecdda1268abb6678bffb1"}
Nov 02 09:46:26.778964 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.778935    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j" event={"ID":"af5227c4-b584-4416-86c2-8a94d61db2cd","Type":"ContainerStarted","Data":"57d2666344faedeb84653003d944885aafca19ec7ac5ddb0d86d7d609763e712"}
Nov 02 09:46:26.779059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.778977    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j" event={"ID":"af5227c4-b584-4416-86c2-8a94d61db2cd","Type":"ContainerStarted","Data":"f7a58574e59d0b254f8c801ae452d89a30c06f1ccd168372ab6fe3ae371987f1"}
Nov 02 09:46:26.779059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.779019    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7360-0dc00d" podUID="1149bbf4-0d14-4db3-b3ad-c43872be93e1" containerName="test-curl-7360-0dc00d" containerID="cri-o://3dfb27ffb224cc1f7e93621775317c0b91bd3ba0cc660ed29a7879abcfa8eaa5" gracePeriod=2
Nov 02 09:46:26.797008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.796953    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7360-90a43e" podStartSLOduration=1.796939798 podStartE2EDuration="1.796939798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:26.794878994 +0000 UTC m=+663.733165793" watchObservedRunningTime="2026-11-02 09:46:26.796939798 +0000 UTC m=+663.735226574"
Nov 02 09:46:26.797343 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:46:26.797318    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7360-0dc00d\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="1149bbf4-0d14-4db3-b3ad-c43872be93e1" pod="opendatahub/test-curl-7360-0dc00d"
Nov 02 09:46:26.814831 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:26.814770    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j" podStartSLOduration=1.814752796 podStartE2EDuration="1.814752796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:26.814020255 +0000 UTC m=+663.752307082" watchObservedRunningTime="2026-11-02 09:46:26.814752796 +0000 UTC m=+663.753039573"
Nov 02 09:46:27.330746 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:27.330699    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8" podUID="2b246a3b-4f40-4174-b9a2-7f0669c1271a" containerName="payload-pre-processing" containerID="cri-o://17e65abb6ecb861ef50043b5b0a709be7288d408a690b9237dbc2e92898dfda6" gracePeriod=25
Nov 02 09:46:27.530889 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:27.530802    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7360-90a43e"]
Nov 02 09:46:27.535859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:27.535821    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7360-90a43e"]
Nov 02 09:46:27.790124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:27.790089    4449 generic.go:356] "Generic (PLEG): container finished" podID="af5227c4-b584-4416-86c2-8a94d61db2cd" containerID="57d2666344faedeb84653003d944885aafca19ec7ac5ddb0d86d7d609763e712" exitCode=0
Nov 02 09:46:27.790338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:27.790150    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j" event={"ID":"af5227c4-b584-4416-86c2-8a94d61db2cd","Type":"ContainerDied","Data":"57d2666344faedeb84653003d944885aafca19ec7ac5ddb0d86d7d609763e712"}
Nov 02 09:46:28.696248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.696200    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"]
Nov 02 09:46:28.696805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.696790    4449 cpu_manager.go:414] "containerMap: removing container" podUID="86ec625d-41e3-40ea-94a2-bb18b3006d3a" containerName="test-curl-7360-90a43e"
Nov 02 09:46:28.696849 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.696806    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ec625d-41e3-40ea-94a2-bb18b3006d3a" containerName="test-curl-7360-90a43e"
Nov 02 09:46:28.696907 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.696898    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ec625d-41e3-40ea-94a2-bb18b3006d3a" containerName="test-curl-7360-90a43e"
Nov 02 09:46:28.699141 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.699123    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.708338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.708316    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-301be8-openshift-default-dockercfg-fxzjg\""
Nov 02 09:46:28.729686 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.729657    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"]
Nov 02 09:46:28.796833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.796797    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7360-90a43e" podUID="86ec625d-41e3-40ea-94a2-bb18b3006d3a" containerName="test-curl-7360-90a43e" containerID="cri-o://d3b8b21a6c9a23aefb503544d033a816d37dbfd80136f3f47b6acf4e54c6dfaa" gracePeriod=2
Nov 02 09:46:28.802516 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:46:28.802469    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7360-90a43e\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="86ec625d-41e3-40ea-94a2-bb18b3006d3a" pod="opendatahub/test-curl-7360-90a43e"
Nov 02 09:46:28.853999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.846429    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-istio-token\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.853999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.846489    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cxh4\" (UniqueName: \"kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-kube-api-access-7cxh4\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.853999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.846538    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-data\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.853999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.846619    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-certs\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.853999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.846710    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef35a63-382b-4978-a07d-197c7775de78-istiod-ca-cert\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.853999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.846761    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef35a63-382b-4978-a07d-197c7775de78-istio-podinfo\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.853999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.846824    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.853999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.846847    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-credential-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.853999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.846917    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-envoy\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.948258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.948189    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-certs\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.948550 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.948527    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef35a63-382b-4978-a07d-197c7775de78-istiod-ca-cert\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.949307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.949291    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef35a63-382b-4978-a07d-197c7775de78-istio-podinfo\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.949448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.949437    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.949525 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.949516    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-credential-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.949638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.949628    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-envoy\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.949721 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.949709    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-istio-token\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.949795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.949785    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cxh4\" (UniqueName: \"kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-kube-api-access-7cxh4\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.949875 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.949865    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-data\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.950138 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.950104    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.950203 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.948668    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-certs\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.950392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.950365    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-credential-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.950706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.949208    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef35a63-382b-4978-a07d-197c7775de78-istiod-ca-cert\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.951414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.951392    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-data\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.953461 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.953435    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-envoy\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.953860 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.953837    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef35a63-382b-4978-a07d-197c7775de78-istio-podinfo\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.964573 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.964524    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cxh4\" (UniqueName: \"kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-kube-api-access-7cxh4\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:28.966725 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:28.966698    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-istio-token\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:29.012101 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.012056    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:29.031023 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.031003    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7360-0dc00d"
Nov 02 09:46:29.033810 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:46:29.033778    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7360-90a43e\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="86ec625d-41e3-40ea-94a2-bb18b3006d3a" pod="opendatahub/test-curl-7360-90a43e"
Nov 02 09:46:29.037808 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.037780    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"
Nov 02 09:46:29.040362 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:46:29.040329    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7360-90a43e\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="86ec625d-41e3-40ea-94a2-bb18b3006d3a" pod="opendatahub/test-curl-7360-90a43e"
Nov 02 09:46:29.151435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.151400    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-openshift-ca\") pod \"af5227c4-b584-4416-86c2-8a94d61db2cd\" (UID: \"af5227c4-b584-4416-86c2-8a94d61db2cd\") "
Nov 02 09:46:29.151608 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.151474    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1149bbf4-0d14-4db3-b3ad-c43872be93e1-kube-api-access-nrd62\" (UniqueName: \"kubernetes.io/projected/1149bbf4-0d14-4db3-b3ad-c43872be93e1-kube-api-access-nrd62\") pod \"1149bbf4-0d14-4db3-b3ad-c43872be93e1\" (UID: \"1149bbf4-0d14-4db3-b3ad-c43872be93e1\") "
Nov 02 09:46:29.151608 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.151552    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5227c4-b584-4416-86c2-8a94d61db2cd-kube-api-access-ncqkz\" (UniqueName: \"kubernetes.io/projected/af5227c4-b584-4416-86c2-8a94d61db2cd-kube-api-access-ncqkz\") pod \"af5227c4-b584-4416-86c2-8a94d61db2cd\" (UID: \"af5227c4-b584-4416-86c2-8a94d61db2cd\") "
Nov 02 09:46:29.151774 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.151676    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-xks-ca\") pod \"af5227c4-b584-4416-86c2-8a94d61db2cd\" (UID: \"af5227c4-b584-4416-86c2-8a94d61db2cd\") "
Nov 02 09:46:29.151836 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.151761    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-openshift-ca" pod "af5227c4-b584-4416-86c2-8a94d61db2cd" (UID: "af5227c4-b584-4416-86c2-8a94d61db2cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:46:29.152094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.152072    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:29.153873 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.153844    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5227c4-b584-4416-86c2-8a94d61db2cd-kube-api-access-ncqkz" pod "af5227c4-b584-4416-86c2-8a94d61db2cd" (UID: "af5227c4-b584-4416-86c2-8a94d61db2cd"). InnerVolumeSpecName "kube-api-access-ncqkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:29.153873 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.153841    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-xks-ca" pod "af5227c4-b584-4416-86c2-8a94d61db2cd" (UID: "af5227c4-b584-4416-86c2-8a94d61db2cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:46:29.154006 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.153900    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1149bbf4-0d14-4db3-b3ad-c43872be93e1-kube-api-access-nrd62" pod "1149bbf4-0d14-4db3-b3ad-c43872be93e1" (UID: "1149bbf4-0d14-4db3-b3ad-c43872be93e1"). InnerVolumeSpecName "kube-api-access-nrd62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:29.207057 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.206976    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1149bbf4-0d14-4db3-b3ad-c43872be93e1" path="/var/lib/kubelet/pods/1149bbf4-0d14-4db3-b3ad-c43872be93e1/volumes"
Nov 02 09:46:29.253078 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.253033    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrd62\" (UniqueName: \"kubernetes.io/projected/1149bbf4-0d14-4db3-b3ad-c43872be93e1-kube-api-access-nrd62\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:29.253078 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.253073    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncqkz\" (UniqueName: \"kubernetes.io/projected/af5227c4-b584-4416-86c2-8a94d61db2cd-kube-api-access-ncqkz\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:29.253078 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.253087    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5227c4-b584-4416-86c2-8a94d61db2cd-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:29.306028 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.305994    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"]
Nov 02 09:46:29.306330 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:46:29.306303    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef35a63_382b_4978_a07d_197c7775de78.slice/crio-3d3c1266eead443d15c5e7240d551a161779208a5b5f6d6e9442af1d36a62675 WatchSource:0}: Error finding container 3d3c1266eead443d15c5e7240d551a161779208a5b5f6d6e9442af1d36a62675: Status 404 returned error can't find the container with id 3d3c1266eead443d15c5e7240d551a161779208a5b5f6d6e9442af1d36a62675
Nov 02 09:46:29.309012 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.308972    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:46:29.309082 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.309060    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:46:29.309121 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.309101    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:46:29.805813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.805776    4449 generic.go:356] "Generic (PLEG): container finished" podID="1149bbf4-0d14-4db3-b3ad-c43872be93e1" containerID="3dfb27ffb224cc1f7e93621775317c0b91bd3ba0cc660ed29a7879abcfa8eaa5" exitCode=137
Nov 02 09:46:29.806231 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.805820    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7360-0dc00d"
Nov 02 09:46:29.806231 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.805854    4449 scope.go:122] "RemoveContainer" containerID="3dfb27ffb224cc1f7e93621775317c0b91bd3ba0cc660ed29a7879abcfa8eaa5"
Nov 02 09:46:29.807583 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.807566    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"
Nov 02 09:46:29.807583 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.807574    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j" event={"ID":"af5227c4-b584-4416-86c2-8a94d61db2cd","Type":"ContainerDied","Data":"f7a58574e59d0b254f8c801ae452d89a30c06f1ccd168372ab6fe3ae371987f1"}
Nov 02 09:46:29.807749 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.807619    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7a58574e59d0b254f8c801ae452d89a30c06f1ccd168372ab6fe3ae371987f1"
Nov 02 09:46:29.809796 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.809771    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk" event={"ID":"7ef35a63-382b-4978-a07d-197c7775de78","Type":"ContainerStarted","Data":"6b64cb0d4613b6a507e71a031ad6fb3647d5a21be29a06a19746e905fe16e23f"}
Nov 02 09:46:29.809903 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.809816    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk" event={"ID":"7ef35a63-382b-4978-a07d-197c7775de78","Type":"ContainerStarted","Data":"3d3c1266eead443d15c5e7240d551a161779208a5b5f6d6e9442af1d36a62675"}
Nov 02 09:46:29.817008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.816990    4449 scope.go:122] "RemoveContainer" containerID="3dfb27ffb224cc1f7e93621775317c0b91bd3ba0cc660ed29a7879abcfa8eaa5"
Nov 02 09:46:29.817331 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:46:29.817310    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfb27ffb224cc1f7e93621775317c0b91bd3ba0cc660ed29a7879abcfa8eaa5\": container with ID starting with 3dfb27ffb224cc1f7e93621775317c0b91bd3ba0cc660ed29a7879abcfa8eaa5 not found: ID does not exist" containerID="3dfb27ffb224cc1f7e93621775317c0b91bd3ba0cc660ed29a7879abcfa8eaa5"
Nov 02 09:46:29.817420 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.817338    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfb27ffb224cc1f7e93621775317c0b91bd3ba0cc660ed29a7879abcfa8eaa5"} err="failed to get container status \"3dfb27ffb224cc1f7e93621775317c0b91bd3ba0cc660ed29a7879abcfa8eaa5\": rpc error: code = NotFound desc = could not find container \"3dfb27ffb224cc1f7e93621775317c0b91bd3ba0cc660ed29a7879abcfa8eaa5\": container with ID starting with 3dfb27ffb224cc1f7e93621775317c0b91bd3ba0cc660ed29a7879abcfa8eaa5 not found: ID does not exist"
Nov 02 09:46:29.836289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:29.836239    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk" podStartSLOduration=1.836209905 podStartE2EDuration="1.836209905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:29.831893849 +0000 UTC m=+666.770180670" watchObservedRunningTime="2026-11-02 09:46:29.836209905 +0000 UTC m=+666.774496685"
Nov 02 09:46:30.012620 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:30.012587    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:30.018490 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:30.018459    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:30.819485 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:30.819457    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:30.820628 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:30.820604    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:31.018074 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:31.018049    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7360-90a43e"
Nov 02 09:46:31.074395 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:31.074306    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ec625d-41e3-40ea-94a2-bb18b3006d3a-kube-api-access-7wj8t\" (UniqueName: \"kubernetes.io/projected/86ec625d-41e3-40ea-94a2-bb18b3006d3a-kube-api-access-7wj8t\") pod \"86ec625d-41e3-40ea-94a2-bb18b3006d3a\" (UID: \"86ec625d-41e3-40ea-94a2-bb18b3006d3a\") "
Nov 02 09:46:31.076621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:31.076586    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ec625d-41e3-40ea-94a2-bb18b3006d3a-kube-api-access-7wj8t" pod "86ec625d-41e3-40ea-94a2-bb18b3006d3a" (UID: "86ec625d-41e3-40ea-94a2-bb18b3006d3a"). InnerVolumeSpecName "kube-api-access-7wj8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:31.175977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:31.175940    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wj8t\" (UniqueName: \"kubernetes.io/projected/86ec625d-41e3-40ea-94a2-bb18b3006d3a-kube-api-access-7wj8t\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:31.206793 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:31.206764    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ec625d-41e3-40ea-94a2-bb18b3006d3a" path="/var/lib/kubelet/pods/86ec625d-41e3-40ea-94a2-bb18b3006d3a/volumes"
Nov 02 09:46:31.827326 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:31.827290    4449 generic.go:356] "Generic (PLEG): container finished" podID="86ec625d-41e3-40ea-94a2-bb18b3006d3a" containerID="d3b8b21a6c9a23aefb503544d033a816d37dbfd80136f3f47b6acf4e54c6dfaa" exitCode=137
Nov 02 09:46:31.827871 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:31.827334    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7360-90a43e"
Nov 02 09:46:31.827871 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:31.827385    4449 scope.go:122] "RemoveContainer" containerID="d3b8b21a6c9a23aefb503544d033a816d37dbfd80136f3f47b6acf4e54c6dfaa"
Nov 02 09:46:31.837891 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:31.837867    4449 scope.go:122] "RemoveContainer" containerID="d3b8b21a6c9a23aefb503544d033a816d37dbfd80136f3f47b6acf4e54c6dfaa"
Nov 02 09:46:31.838309 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:46:31.838276    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b8b21a6c9a23aefb503544d033a816d37dbfd80136f3f47b6acf4e54c6dfaa\": container with ID starting with d3b8b21a6c9a23aefb503544d033a816d37dbfd80136f3f47b6acf4e54c6dfaa not found: ID does not exist" containerID="d3b8b21a6c9a23aefb503544d033a816d37dbfd80136f3f47b6acf4e54c6dfaa"
Nov 02 09:46:31.838513 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:31.838319    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b8b21a6c9a23aefb503544d033a816d37dbfd80136f3f47b6acf4e54c6dfaa"} err="failed to get container status \"d3b8b21a6c9a23aefb503544d033a816d37dbfd80136f3f47b6acf4e54c6dfaa\": rpc error: code = NotFound desc = could not find container \"d3b8b21a6c9a23aefb503544d033a816d37dbfd80136f3f47b6acf4e54c6dfaa\": container with ID starting with d3b8b21a6c9a23aefb503544d033a816d37dbfd80136f3f47b6acf4e54c6dfaa not found: ID does not exist"
Nov 02 09:46:32.014832 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.014795    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"]
Nov 02 09:46:32.015401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.015386    4449 cpu_manager.go:414] "containerMap: removing container" podUID="af5227c4-b584-4416-86c2-8a94d61db2cd" containerName="revoke-keys"
Nov 02 09:46:32.015481 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.015403    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af5227c4-b584-4416-86c2-8a94d61db2cd" containerName="revoke-keys"
Nov 02 09:46:32.015541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.015528    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="af5227c4-b584-4416-86c2-8a94d61db2cd" containerName="revoke-keys"
Nov 02 09:46:32.019813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.019792    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.031575 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.031477    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"]
Nov 02 09:46:32.031575 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.031559    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"openshift-service-ca.crt\""
Nov 02 09:46:32.031862 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.031594    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"default-dockercfg-xf5qm\""
Nov 02 09:46:32.031862 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.031672    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"kube-root-ca.crt\""
Nov 02 09:46:32.031862 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.031741    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"distinct-2-w3-a110ef-kserve-self-signed-certs\""
Nov 02 09:46:32.087511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.087419    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3033ba-0513-428c-9380-50d8f9f3bca8-tls-certs\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.087511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.087497    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-tmp-dir\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.087717 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.087622    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-dshm\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.087757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.087733    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjcq\" (UniqueName: \"kubernetes.io/projected/cb3033ba-0513-428c-9380-50d8f9f3bca8-kube-api-access-8sjcq\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.087810 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.087774    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-model-cache\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.087882 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.087862    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-home\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.195240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.189371    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3033ba-0513-428c-9380-50d8f9f3bca8-tls-certs\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.195240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.189475    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-tmp-dir\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.195240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.189557    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-dshm\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.195240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.189633    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjcq\" (UniqueName: \"kubernetes.io/projected/cb3033ba-0513-428c-9380-50d8f9f3bca8-kube-api-access-8sjcq\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.195240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.189670    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-model-cache\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.195240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.189709    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-home\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.195240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.190318    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-home\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.195240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.192093    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-tmp-dir\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.195240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.192356    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-model-cache\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.197502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.197469    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-dshm\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.208271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.208230    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3033ba-0513-428c-9380-50d8f9f3bca8-tls-certs\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.212671 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.212625    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjcq\" (UniqueName: \"kubernetes.io/projected/cb3033ba-0513-428c-9380-50d8f9f3bca8-kube-api-access-8sjcq\") pod \"distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.337821 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.337726    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:32.620417 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.620385    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"]
Nov 02 09:46:32.632069 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:46:32.622027    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb3033ba_0513_428c_9380_50d8f9f3bca8.slice/crio-39b7c914d8972c5e0d637fd9f76dba4612ddf6658b9048a38d02439a6551c61b WatchSource:0}: Error finding container 39b7c914d8972c5e0d637fd9f76dba4612ddf6658b9048a38d02439a6551c61b: Status 404 returned error can't find the container with id 39b7c914d8972c5e0d637fd9f76dba4612ddf6658b9048a38d02439a6551c61b
Nov 02 09:46:32.837293 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.837253    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" event={"ID":"cb3033ba-0513-428c-9380-50d8f9f3bca8","Type":"ContainerStarted","Data":"ac46986bdef3856072a0ab53c7e690a75d9755efb227605139d3ea1dec80bbcc"}
Nov 02 09:46:32.837757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.837304    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" event={"ID":"cb3033ba-0513-428c-9380-50d8f9f3bca8","Type":"ContainerStarted","Data":"39b7c914d8972c5e0d637fd9f76dba4612ddf6658b9048a38d02439a6551c61b"}
Nov 02 09:46:32.857331 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:32.857259    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podStartSLOduration=1.857239662 podStartE2EDuration="1.857239662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:32.857105055 +0000 UTC m=+669.795391834" watchObservedRunningTime="2026-11-02 09:46:32.857239662 +0000 UTC m=+669.795526441"
Nov 02 09:46:33.337885 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.337846    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:33.455447 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.455410    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"]
Nov 02 09:46:33.459539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.459514    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"
Nov 02 09:46:33.467465 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.467440    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-301be8\""
Nov 02 09:46:33.471934 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.471909    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"]
Nov 02 09:46:33.497558 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.497515    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"]
Nov 02 09:46:33.502996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.502966    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"
Nov 02 09:46:33.506047 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.505442    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"]
Nov 02 09:46:33.510316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.509693    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-301be8-dockercfg-z8djr\""
Nov 02 09:46:33.619732 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.607670    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqsdn\" (UniqueName: \"kubernetes.io/projected/094a6ff2-c320-45e3-839a-9ba70ec6da87-kube-api-access-rqsdn\") pod \"payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll\" (UID: \"094a6ff2-c320-45e3-839a-9ba70ec6da87\") " pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"
Nov 02 09:46:33.619732 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.607839    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvcfh\" (UniqueName: \"kubernetes.io/projected/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-kube-api-access-vvcfh\") pod \"payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4\" (UID: \"f1a365ff-43d5-400d-80d6-0cd798f2f7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"
Nov 02 09:46:33.619732 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.607941    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4\" (UID: \"f1a365ff-43d5-400d-80d6-0cd798f2f7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"
Nov 02 09:46:33.619732 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.607969    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094a6ff2-c320-45e3-839a-9ba70ec6da87-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll\" (UID: \"094a6ff2-c320-45e3-839a-9ba70ec6da87\") " pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"
Nov 02 09:46:33.709498 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.709458    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvcfh\" (UniqueName: \"kubernetes.io/projected/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-kube-api-access-vvcfh\") pod \"payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4\" (UID: \"f1a365ff-43d5-400d-80d6-0cd798f2f7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"
Nov 02 09:46:33.709842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.709823    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4\" (UID: \"f1a365ff-43d5-400d-80d6-0cd798f2f7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"
Nov 02 09:46:33.709960 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.709944    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094a6ff2-c320-45e3-839a-9ba70ec6da87-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll\" (UID: \"094a6ff2-c320-45e3-839a-9ba70ec6da87\") " pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"
Nov 02 09:46:33.710134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.710119    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqsdn\" (UniqueName: \"kubernetes.io/projected/094a6ff2-c320-45e3-839a-9ba70ec6da87-kube-api-access-rqsdn\") pod \"payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll\" (UID: \"094a6ff2-c320-45e3-839a-9ba70ec6da87\") " pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"
Nov 02 09:46:33.710933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.710898    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4\" (UID: \"f1a365ff-43d5-400d-80d6-0cd798f2f7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"
Nov 02 09:46:33.711527 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.711504    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094a6ff2-c320-45e3-839a-9ba70ec6da87-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll\" (UID: \"094a6ff2-c320-45e3-839a-9ba70ec6da87\") " pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"
Nov 02 09:46:33.723173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.723126    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqsdn\" (UniqueName: \"kubernetes.io/projected/094a6ff2-c320-45e3-839a-9ba70ec6da87-kube-api-access-rqsdn\") pod \"payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll\" (UID: \"094a6ff2-c320-45e3-839a-9ba70ec6da87\") " pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"
Nov 02 09:46:33.730708 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.730655    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvcfh\" (UniqueName: \"kubernetes.io/projected/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-kube-api-access-vvcfh\") pod \"payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4\" (UID: \"f1a365ff-43d5-400d-80d6-0cd798f2f7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"
Nov 02 09:46:33.778505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.778457    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"
Nov 02 09:46:33.822957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:33.822915    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"
Nov 02 09:46:34.079771 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:46:34.079511    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094a6ff2_c320_45e3_839a_9ba70ec6da87.slice/crio-16ee763ca41fb4b6a7ff235fd9714d6ab6d545795b19b4de917bbbeca8085ca7 WatchSource:0}: Error finding container 16ee763ca41fb4b6a7ff235fd9714d6ab6d545795b19b4de917bbbeca8085ca7: Status 404 returned error can't find the container with id 16ee763ca41fb4b6a7ff235fd9714d6ab6d545795b19b4de917bbbeca8085ca7
Nov 02 09:46:34.079771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:34.079628    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"]
Nov 02 09:46:34.189451 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:34.189406    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"]
Nov 02 09:46:34.885189 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:34.885154    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4" event={"ID":"f1a365ff-43d5-400d-80d6-0cd798f2f7bf","Type":"ContainerStarted","Data":"62f07ce53eadb44da1b295ba773b1cc43769adde6b287a12278dee3f7030a3b4"}
Nov 02 09:46:34.885393 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:34.885203    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4" event={"ID":"f1a365ff-43d5-400d-80d6-0cd798f2f7bf","Type":"ContainerStarted","Data":"20a596f02629fc1a3e29f169de1fa7c88230088e45b2a16a21f4bf933c3b816c"}
Nov 02 09:46:34.885393 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:34.885280    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"
Nov 02 09:46:34.887058 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:34.887030    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll" event={"ID":"094a6ff2-c320-45e3-839a-9ba70ec6da87","Type":"ContainerStarted","Data":"9064bddc66b23dfde817a3367c46c126568525071e56f8f23799b1632f6149d2"}
Nov 02 09:46:34.887188 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:34.887075    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll" event={"ID":"094a6ff2-c320-45e3-839a-9ba70ec6da87","Type":"ContainerStarted","Data":"16ee763ca41fb4b6a7ff235fd9714d6ab6d545795b19b4de917bbbeca8085ca7"}
Nov 02 09:46:34.887188 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:34.887125    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"
Nov 02 09:46:34.904200 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:34.904150    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4" podStartSLOduration=1.904138762 podStartE2EDuration="1.904138762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:34.902928285 +0000 UTC m=+671.841215091" watchObservedRunningTime="2026-11-02 09:46:34.904138762 +0000 UTC m=+671.842425535"
Nov 02 09:46:34.922783 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:34.922738    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll" podStartSLOduration=1.922725694 podStartE2EDuration="1.922725694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:34.920068394 +0000 UTC m=+671.858355182" watchObservedRunningTime="2026-11-02 09:46:34.922725694 +0000 UTC m=+671.861012487"
Nov 02 09:46:36.179435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:36.179397    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8" podUID="2b246a3b-4f40-4174-b9a2-7f0669c1271a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:46:36.548472 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:46:36.548423    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:46:38.170996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.170958    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"]
Nov 02 09:46:38.175040 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.175021    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"
Nov 02 09:46:38.180523 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.180500    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:46:38.180757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.180729    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:46:38.181266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.181062    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:46:38.181409 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.180518    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:46:38.184105 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.184086    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"]
Nov 02 09:46:38.272885 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.272850    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc5lv\" (UniqueName: \"kubernetes.io/projected/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-kube-api-access-lc5lv\") pod \"maas-api-revoke-sub-9c06d57b-7xsl4\" (UID: \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"
Nov 02 09:46:38.273125 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.272925    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c06d57b-7xsl4\" (UID: \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"
Nov 02 09:46:38.273125 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.273047    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c06d57b-7xsl4\" (UID: \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"
Nov 02 09:46:38.374111 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.374069    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc5lv\" (UniqueName: \"kubernetes.io/projected/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-kube-api-access-lc5lv\") pod \"maas-api-revoke-sub-9c06d57b-7xsl4\" (UID: \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"
Nov 02 09:46:38.374306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.374134    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c06d57b-7xsl4\" (UID: \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"
Nov 02 09:46:38.374306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.374175    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c06d57b-7xsl4\" (UID: \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"
Nov 02 09:46:38.374817 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.374793    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c06d57b-7xsl4\" (UID: \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"
Nov 02 09:46:38.376750 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.376724    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c06d57b-7xsl4\" (UID: \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"
Nov 02 09:46:38.389297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.389269    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc5lv\" (UniqueName: \"kubernetes.io/projected/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-kube-api-access-lc5lv\") pod \"maas-api-revoke-sub-9c06d57b-7xsl4\" (UID: \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"
Nov 02 09:46:38.491371 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.491300    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"
Nov 02 09:46:38.606463 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.606429    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"]
Nov 02 09:46:38.606924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.606702    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw" podUID="1fe7331b-6724-4406-82bc-814e9dc8179c" containerName="istio-proxy" containerID="cri-o://5e12f7b82a63d2d1dd12fbae2b20650fae5634af4d86024cb26bbea4e495ec72" gracePeriod=30
Nov 02 09:46:38.796259 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.796229    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"]
Nov 02 09:46:38.797305 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:46:38.797276    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48777f59_1ff1_4d26_a615_22c1f9ee1ff7.slice/crio-4c13340b5edcfe8a3aaa43f6eceeef76860b3a6fdf41fc3538faff284a87b41c WatchSource:0}: Error finding container 4c13340b5edcfe8a3aaa43f6eceeef76860b3a6fdf41fc3538faff284a87b41c: Status 404 returned error can't find the container with id 4c13340b5edcfe8a3aaa43f6eceeef76860b3a6fdf41fc3538faff284a87b41c
Nov 02 09:46:38.922299 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:38.922258    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4" event={"ID":"48777f59-1ff1-4d26-a615-22c1f9ee1ff7","Type":"ContainerStarted","Data":"4c13340b5edcfe8a3aaa43f6eceeef76860b3a6fdf41fc3538faff284a87b41c"}
Nov 02 09:46:39.938387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:39.936378    4449 generic.go:356] "Generic (PLEG): container finished" podID="48777f59-1ff1-4d26-a615-22c1f9ee1ff7" containerID="1a4e270bec975d6f28cfe23954c445a512a7559a271b40b066bebd9411b7b15c" exitCode=0
Nov 02 09:46:39.938387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:39.936776    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4" event={"ID":"48777f59-1ff1-4d26-a615-22c1f9ee1ff7","Type":"ContainerDied","Data":"1a4e270bec975d6f28cfe23954c445a512a7559a271b40b066bebd9411b7b15c"}
Nov 02 09:46:41.095639 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:41.095615    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"
Nov 02 09:46:41.207154 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:41.207073    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-openshift-ca\") pod \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\" (UID: \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\") "
Nov 02 09:46:41.207321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:41.207248    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-kube-api-access-lc5lv\" (UniqueName: \"kubernetes.io/projected/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-kube-api-access-lc5lv\") pod \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\" (UID: \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\") "
Nov 02 09:46:41.207321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:41.207268    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-xks-ca\") pod \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\" (UID: \"48777f59-1ff1-4d26-a615-22c1f9ee1ff7\") "
Nov 02 09:46:41.207514 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:41.207487    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-openshift-ca" pod "48777f59-1ff1-4d26-a615-22c1f9ee1ff7" (UID: "48777f59-1ff1-4d26-a615-22c1f9ee1ff7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:46:41.207631 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:41.207609    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:41.209542 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:41.209517    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-xks-ca" pod "48777f59-1ff1-4d26-a615-22c1f9ee1ff7" (UID: "48777f59-1ff1-4d26-a615-22c1f9ee1ff7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:46:41.209797 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:41.209780    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-kube-api-access-lc5lv" pod "48777f59-1ff1-4d26-a615-22c1f9ee1ff7" (UID: "48777f59-1ff1-4d26-a615-22c1f9ee1ff7"). InnerVolumeSpecName "kube-api-access-lc5lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:41.308529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:41.308497    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc5lv\" (UniqueName: \"kubernetes.io/projected/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-kube-api-access-lc5lv\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:41.308529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:41.308527    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48777f59-1ff1-4d26-a615-22c1f9ee1ff7-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:41.951524 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:41.951486    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4" event={"ID":"48777f59-1ff1-4d26-a615-22c1f9ee1ff7","Type":"ContainerDied","Data":"4c13340b5edcfe8a3aaa43f6eceeef76860b3a6fdf41fc3538faff284a87b41c"}
Nov 02 09:46:41.951524 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:41.951514    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"
Nov 02 09:46:41.951694 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:41.951546    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c13340b5edcfe8a3aaa43f6eceeef76860b3a6fdf41fc3538faff284a87b41c"
Nov 02 09:46:42.338522 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:42.338489    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:42.354498 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:42.354471    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:42.970669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:42.970639    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:46:43.974471 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:43.974436    4449 generic.go:356] "Generic (PLEG): container finished" podID="1fe7331b-6724-4406-82bc-814e9dc8179c" containerID="5e12f7b82a63d2d1dd12fbae2b20650fae5634af4d86024cb26bbea4e495ec72" exitCode=0
Nov 02 09:46:43.976415 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:43.976293    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw" event={"ID":"1fe7331b-6724-4406-82bc-814e9dc8179c","Type":"ContainerDied","Data":"5e12f7b82a63d2d1dd12fbae2b20650fae5634af4d86024cb26bbea4e495ec72"}
Nov 02 09:46:43.976625 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:43.976603    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw" event={"ID":"1fe7331b-6724-4406-82bc-814e9dc8179c","Type":"ContainerDied","Data":"81f82ce3926f6436c433e0cd9756230f624a0681d52c77743959a367300343b7"}
Nov 02 09:46:43.976811 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:43.976795    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81f82ce3926f6436c433e0cd9756230f624a0681d52c77743959a367300343b7"
Nov 02 09:46:43.990469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:43.990440    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:46:44.148416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.148362    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-socket\") pod \"1fe7331b-6724-4406-82bc-814e9dc8179c\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") "
Nov 02 09:46:44.148633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.148461    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-credential-socket\") pod \"1fe7331b-6724-4406-82bc-814e9dc8179c\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") "
Nov 02 09:46:44.148633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.148535    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-data\") pod \"1fe7331b-6724-4406-82bc-814e9dc8179c\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") "
Nov 02 09:46:44.148633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.148577    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-envoy\") pod \"1fe7331b-6724-4406-82bc-814e9dc8179c\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") "
Nov 02 09:46:44.148813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.148643    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-token\" (UniqueName: \"kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-token\") pod \"1fe7331b-6724-4406-82bc-814e9dc8179c\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") "
Nov 02 09:46:44.148813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.148666    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-socket" pod "1fe7331b-6724-4406-82bc-814e9dc8179c" (UID: "1fe7331b-6724-4406-82bc-814e9dc8179c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:44.148813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.148736    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-kube-api-access-rvhg2\" (UniqueName: \"kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-kube-api-access-rvhg2\") pod \"1fe7331b-6724-4406-82bc-814e9dc8179c\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") "
Nov 02 09:46:44.148813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.148767    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-credential-socket" pod "1fe7331b-6724-4406-82bc-814e9dc8179c" (UID: "1fe7331b-6724-4406-82bc-814e9dc8179c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:44.149033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.148827    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-data" pod "1fe7331b-6724-4406-82bc-814e9dc8179c" (UID: "1fe7331b-6724-4406-82bc-814e9dc8179c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:44.149033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.148841    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe7331b-6724-4406-82bc-814e9dc8179c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe7331b-6724-4406-82bc-814e9dc8179c-istiod-ca-cert\") pod \"1fe7331b-6724-4406-82bc-814e9dc8179c\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") "
Nov 02 09:46:44.149033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.148893    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-podinfo\") pod \"1fe7331b-6724-4406-82bc-814e9dc8179c\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") "
Nov 02 09:46:44.149033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.148946    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-certs\") pod \"1fe7331b-6724-4406-82bc-814e9dc8179c\" (UID: \"1fe7331b-6724-4406-82bc-814e9dc8179c\") "
Nov 02 09:46:44.149466 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.149419    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe7331b-6724-4406-82bc-814e9dc8179c-istiod-ca-cert" pod "1fe7331b-6724-4406-82bc-814e9dc8179c" (UID: "1fe7331b-6724-4406-82bc-814e9dc8179c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:46:44.149787 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.149757    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-certs" pod "1fe7331b-6724-4406-82bc-814e9dc8179c" (UID: "1fe7331b-6724-4406-82bc-814e9dc8179c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:44.150163 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.150138    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.150278 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.150169    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.150278 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.150184    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.150278 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.150199    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe7331b-6724-4406-82bc-814e9dc8179c-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.150278 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.150255    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.151557 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.151529    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-token" pod "1fe7331b-6724-4406-82bc-814e9dc8179c" (UID: "1fe7331b-6724-4406-82bc-814e9dc8179c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:44.151845 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.151819    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-podinfo" pod "1fe7331b-6724-4406-82bc-814e9dc8179c" (UID: "1fe7331b-6724-4406-82bc-814e9dc8179c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:46:44.154168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.153256    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-kube-api-access-rvhg2" pod "1fe7331b-6724-4406-82bc-814e9dc8179c" (UID: "1fe7331b-6724-4406-82bc-814e9dc8179c"). InnerVolumeSpecName "kube-api-access-rvhg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:44.155612 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.155321    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-envoy" pod "1fe7331b-6724-4406-82bc-814e9dc8179c" (UID: "1fe7331b-6724-4406-82bc-814e9dc8179c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:44.253167 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.253001    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.253167 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.253044    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.253167 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.253061    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvhg2\" (UniqueName: \"kubernetes.io/projected/1fe7331b-6724-4406-82bc-814e9dc8179c-kube-api-access-rvhg2\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.253167 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.253080    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe7331b-6724-4406-82bc-814e9dc8179c-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.999331 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:44.999290    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"
Nov 02 09:46:45.037149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:45.037106    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"]
Nov 02 09:46:45.042497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:45.042470    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-lvtlw"]
Nov 02 09:46:45.213862 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:45.213734    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe7331b-6724-4406-82bc-814e9dc8179c" path="/var/lib/kubelet/pods/1fe7331b-6724-4406-82bc-814e9dc8179c/volumes"
Nov 02 09:46:45.896828 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:45.896801    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"
Nov 02 09:46:45.897022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:45.896863    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"
Nov 02 09:46:46.179789 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:46.179695    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8" podUID="2b246a3b-4f40-4174-b9a2-7f0669c1271a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:46:47.684073 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:47.684040    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"
Nov 02 09:46:47.729559 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:47.729530    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b246a3b-4f40-4174-b9a2-7f0669c1271a-kube-api-access-whnw4\" (UniqueName: \"kubernetes.io/projected/2b246a3b-4f40-4174-b9a2-7f0669c1271a-kube-api-access-whnw4\") pod \"2b246a3b-4f40-4174-b9a2-7f0669c1271a\" (UID: \"2b246a3b-4f40-4174-b9a2-7f0669c1271a\") "
Nov 02 09:46:47.729835 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:47.729810    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b246a3b-4f40-4174-b9a2-7f0669c1271a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b246a3b-4f40-4174-b9a2-7f0669c1271a-plugins-config-volume\") pod \"2b246a3b-4f40-4174-b9a2-7f0669c1271a\" (UID: \"2b246a3b-4f40-4174-b9a2-7f0669c1271a\") "
Nov 02 09:46:47.730312 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:47.730281    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b246a3b-4f40-4174-b9a2-7f0669c1271a-plugins-config-volume" pod "2b246a3b-4f40-4174-b9a2-7f0669c1271a" (UID: "2b246a3b-4f40-4174-b9a2-7f0669c1271a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:46:47.730497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:47.730482    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b246a3b-4f40-4174-b9a2-7f0669c1271a-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:47.731809 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:47.731790    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b246a3b-4f40-4174-b9a2-7f0669c1271a-kube-api-access-whnw4" pod "2b246a3b-4f40-4174-b9a2-7f0669c1271a" (UID: "2b246a3b-4f40-4174-b9a2-7f0669c1271a"). InnerVolumeSpecName "kube-api-access-whnw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:47.831307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:47.831270    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whnw4\" (UniqueName: \"kubernetes.io/projected/2b246a3b-4f40-4174-b9a2-7f0669c1271a-kube-api-access-whnw4\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:48.020604 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:48.020503    4449 generic.go:356] "Generic (PLEG): container finished" podID="2b246a3b-4f40-4174-b9a2-7f0669c1271a" containerID="17e65abb6ecb861ef50043b5b0a709be7288d408a690b9237dbc2e92898dfda6" exitCode=0
Nov 02 09:46:48.020604 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:48.020545    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"
Nov 02 09:46:48.020604 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:48.020575    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8" event={"ID":"2b246a3b-4f40-4174-b9a2-7f0669c1271a","Type":"ContainerDied","Data":"17e65abb6ecb861ef50043b5b0a709be7288d408a690b9237dbc2e92898dfda6"}
Nov 02 09:46:48.020844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:48.020635    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8" event={"ID":"2b246a3b-4f40-4174-b9a2-7f0669c1271a","Type":"ContainerDied","Data":"d218a44e1cb74e1ba432831c5ae059999048c597c7ae3d1c586c5efb823099ce"}
Nov 02 09:46:48.020844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:48.020655    4449 scope.go:122] "RemoveContainer" containerID="17e65abb6ecb861ef50043b5b0a709be7288d408a690b9237dbc2e92898dfda6"
Nov 02 09:46:48.032114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:48.032097    4449 scope.go:122] "RemoveContainer" containerID="17e65abb6ecb861ef50043b5b0a709be7288d408a690b9237dbc2e92898dfda6"
Nov 02 09:46:48.033356 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:46:48.033333    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e65abb6ecb861ef50043b5b0a709be7288d408a690b9237dbc2e92898dfda6\": container with ID starting with 17e65abb6ecb861ef50043b5b0a709be7288d408a690b9237dbc2e92898dfda6 not found: ID does not exist" containerID="17e65abb6ecb861ef50043b5b0a709be7288d408a690b9237dbc2e92898dfda6"
Nov 02 09:46:48.033512 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:48.033378    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e65abb6ecb861ef50043b5b0a709be7288d408a690b9237dbc2e92898dfda6"} err="failed to get container status \"17e65abb6ecb861ef50043b5b0a709be7288d408a690b9237dbc2e92898dfda6\": rpc error: code = NotFound desc = could not find container \"17e65abb6ecb861ef50043b5b0a709be7288d408a690b9237dbc2e92898dfda6\": container with ID starting with 17e65abb6ecb861ef50043b5b0a709be7288d408a690b9237dbc2e92898dfda6 not found: ID does not exist"
Nov 02 09:46:48.071247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:48.070264    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"]
Nov 02 09:46:48.078057 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:48.078024    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-000ffb-76b97879b7-lbxd8"]
Nov 02 09:46:49.206729 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:49.206693    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b246a3b-4f40-4174-b9a2-7f0669c1271a" path="/var/lib/kubelet/pods/2b246a3b-4f40-4174-b9a2-7f0669c1271a/volumes"
Nov 02 09:46:50.707922 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:50.707830    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"]
Nov 02 09:46:50.724267 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:50.724231    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"]
Nov 02 09:46:52.430071 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.430037    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"]
Nov 02 09:46:52.430657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.430636    4449 cpu_manager.go:414] "containerMap: removing container" podUID="2b246a3b-4f40-4174-b9a2-7f0669c1271a" containerName="payload-pre-processing"
Nov 02 09:46:52.430657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.430655    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b246a3b-4f40-4174-b9a2-7f0669c1271a" containerName="payload-pre-processing"
Nov 02 09:46:52.430767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.430665    4449 cpu_manager.go:414] "containerMap: removing container" podUID="48777f59-1ff1-4d26-a615-22c1f9ee1ff7" containerName="revoke-keys"
Nov 02 09:46:52.430767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.430670    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48777f59-1ff1-4d26-a615-22c1f9ee1ff7" containerName="revoke-keys"
Nov 02 09:46:52.430767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.430710    4449 cpu_manager.go:414] "containerMap: removing container" podUID="1fe7331b-6724-4406-82bc-814e9dc8179c" containerName="istio-proxy"
Nov 02 09:46:52.430767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.430715    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe7331b-6724-4406-82bc-814e9dc8179c" containerName="istio-proxy"
Nov 02 09:46:52.430887 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.430800    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe7331b-6724-4406-82bc-814e9dc8179c" containerName="istio-proxy"
Nov 02 09:46:52.430887 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.430810    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b246a3b-4f40-4174-b9a2-7f0669c1271a" containerName="payload-pre-processing"
Nov 02 09:46:52.430887 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.430818    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="48777f59-1ff1-4d26-a615-22c1f9ee1ff7" containerName="revoke-keys"
Nov 02 09:46:52.435571 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.435545    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.447302 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.447271    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"]
Nov 02 09:46:52.504050 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.503991    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.504258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.504121    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.504258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.504183    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.504412 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.504259    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.504412 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.504296    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.504412 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.504394    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x7f2\" (UniqueName: \"kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-kube-api-access-4x7f2\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.504592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.504469    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.504592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.504499    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.504592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.504521    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606019 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.605969    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606019 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.606012    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.606143    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.606211    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.606316    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x7f2\" (UniqueName: \"kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-kube-api-access-4x7f2\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606503 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.606361    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606503 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.606392    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606503 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.606418    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606503 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.606437    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.606609    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.606734    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606848 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.606816    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606848 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.606835    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.606998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.606975    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.608914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.608892    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.609026 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.608911    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.619192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.619160    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.619340 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.619320    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x7f2\" (UniqueName: \"kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-kube-api-access-4x7f2\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:52.764318 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:52.764205    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:53.065494 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.065456    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"]
Nov 02 09:46:53.066669 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:46:53.066614    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0663f6_24d9_42e1_b69d_d1c73b24fa6b.slice/crio-16d7fd1b833348ea1de70cd32dbd22adaf8b6d5ac0b084767552d0ccc39b4239 WatchSource:0}: Error finding container 16d7fd1b833348ea1de70cd32dbd22adaf8b6d5ac0b084767552d0ccc39b4239: Status 404 returned error can't find the container with id 16d7fd1b833348ea1de70cd32dbd22adaf8b6d5ac0b084767552d0ccc39b4239
Nov 02 09:46:53.072123 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.072081    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:46:53.072285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.072178    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:46:53.072285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.072246    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:46:53.116128 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.116097    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"]
Nov 02 09:46:53.122407 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.122379    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.127922 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.127901    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8743a5-openshift-default-dockercfg-6pdx6\""
Nov 02 09:46:53.135163 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.135139    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"]
Nov 02 09:46:53.215086 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.214753    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5928113f-5ce3-4403-b390-d52a356939af-istiod-ca-cert\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.215660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.215124    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.215660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.215289    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-envoy\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.215660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.215375    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5928113f-5ce3-4403-b390-d52a356939af-istio-podinfo\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.215660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.215484    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-data\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.215660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.215561    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4nqj\" (UniqueName: \"kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-kube-api-access-d4nqj\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.215660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.215621    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-istio-token\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.243673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.215684    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-certs\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.243673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.215776    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-credential-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.317323    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4nqj\" (UniqueName: \"kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-kube-api-access-d4nqj\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.317382    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-istio-token\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.317415    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-certs\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.317471    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-credential-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.317625    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5928113f-5ce3-4403-b390-d52a356939af-istiod-ca-cert\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.317672    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.317730    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-envoy\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.317779    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5928113f-5ce3-4403-b390-d52a356939af-istio-podinfo\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.317862    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-data\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.318375    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-data\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.318663    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-credential-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.319415    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-certs\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.319516    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5928113f-5ce3-4403-b390-d52a356939af-istiod-ca-cert\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.322288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.320423    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.326516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.326489    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-envoy\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.326956 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.326922    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5928113f-5ce3-4403-b390-d52a356939af-istio-podinfo\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.333054 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.333021    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-istio-token\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.334098 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.334067    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4nqj\" (UniqueName: \"kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-kube-api-access-d4nqj\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.454957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.454911    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:53.755121 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.755093    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"]
Nov 02 09:46:53.757540 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:46:53.757501    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5928113f_5ce3_4403_b390_d52a356939af.slice/crio-464837686f2ed64a6ac24f12ea5ca26dc1819159c8c8e5aed284c109a033e895 WatchSource:0}: Error finding container 464837686f2ed64a6ac24f12ea5ca26dc1819159c8c8e5aed284c109a033e895: Status 404 returned error can't find the container with id 464837686f2ed64a6ac24f12ea5ca26dc1819159c8c8e5aed284c109a033e895
Nov 02 09:46:53.761015 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.760976    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:46:53.761102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.761069    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:46:53.761143 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:53.761112    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:46:54.088304 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:54.087902    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" event={"ID":"5928113f-5ce3-4403-b390-d52a356939af","Type":"ContainerStarted","Data":"d962a00c6aa93312933d4291fdbf5c290952111d56e3b4657c157b35e7af97c7"}
Nov 02 09:46:54.088304 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:54.087970    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" event={"ID":"5928113f-5ce3-4403-b390-d52a356939af","Type":"ContainerStarted","Data":"464837686f2ed64a6ac24f12ea5ca26dc1819159c8c8e5aed284c109a033e895"}
Nov 02 09:46:54.092289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:54.091506    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" event={"ID":"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b","Type":"ContainerStarted","Data":"cc823ff680fc5ec8259e6d3c1d6bd3e5fc32ebdc45e72b5973902af52dc77b2c"}
Nov 02 09:46:54.092289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:54.091566    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" event={"ID":"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b","Type":"ContainerStarted","Data":"16d7fd1b833348ea1de70cd32dbd22adaf8b6d5ac0b084767552d0ccc39b4239"}
Nov 02 09:46:54.116027 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:54.115966    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" podStartSLOduration=1.115947648 podStartE2EDuration="1.115947648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:54.111622352 +0000 UTC m=+691.049909151" watchObservedRunningTime="2026-11-02 09:46:54.115947648 +0000 UTC m=+691.054234425"
Nov 02 09:46:54.137066 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:54.136997    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" podStartSLOduration=2.136975348 podStartE2EDuration="2.136975348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:54.133915672 +0000 UTC m=+691.072202450" watchObservedRunningTime="2026-11-02 09:46:54.136975348 +0000 UTC m=+691.075262128"
Nov 02 09:46:54.468300 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:54.458612    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:54.480614 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:54.480581    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:54.774306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:54.769683    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:46:54.775731 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:54.775695    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 09:46:54.775874 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:54.775766    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" podUID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 09:46:55.104869 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:55.104835    4449 generic.go:356] "Generic (PLEG): container finished" podID="7ef35a63-382b-4978-a07d-197c7775de78" containerID="6b64cb0d4613b6a507e71a031ad6fb3647d5a21be29a06a19746e905fe16e23f" exitCode=0
Nov 02 09:46:55.105146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:55.104891    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk" event={"ID":"7ef35a63-382b-4978-a07d-197c7775de78","Type":"ContainerDied","Data":"6b64cb0d4613b6a507e71a031ad6fb3647d5a21be29a06a19746e905fe16e23f"}
Nov 02 09:46:55.106184 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:55.106117    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:55.108204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:55.108178    4449 scope.go:122] "RemoveContainer" containerID="6b64cb0d4613b6a507e71a031ad6fb3647d5a21be29a06a19746e905fe16e23f"
Nov 02 09:46:55.112373 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:55.112281    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:46:55.113775 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:55.112822    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:46:55.113775 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:55.112939    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:46:55.113775 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:55.113006    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:46:55.708519 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:55.708445    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv" podUID="09ec8375-f77f-4b8f-a21c-9303968468ca" containerName="payload-processing" containerID="cri-o://345b6f469a814342ef530b035b42b4dbf711cd303c3506d5bbd5ae894a0bcc1e" gracePeriod=25
Nov 02 09:46:55.725113 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:55.725047    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n" podUID="7f72acb2-73e1-4e88-8639-de0fb5ddce30" containerName="payload-pre-processing" containerID="cri-o://ea657f63575b49626dbe683db4b461926e65d695c5d11e105daec88cde15b106" gracePeriod=25
Nov 02 09:46:55.771723 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:55.771680    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 09:46:55.772011 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:55.771978    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" podUID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 09:46:56.129794 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.129759    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"
Nov 02 09:46:56.133246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.132871    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk" event={"ID":"7ef35a63-382b-4978-a07d-197c7775de78","Type":"ContainerStarted","Data":"16448d7ef350e8c89b0630a4528613835d07d4bc575d923a09e20f229c93e107"}
Nov 02 09:46:56.137844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.137810    4449 generic.go:356] "Generic (PLEG): container finished" podID="7f72acb2-73e1-4e88-8639-de0fb5ddce30" containerID="ea657f63575b49626dbe683db4b461926e65d695c5d11e105daec88cde15b106" exitCode=0
Nov 02 09:46:56.138019 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.137918    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n" event={"ID":"7f72acb2-73e1-4e88-8639-de0fb5ddce30","Type":"ContainerDied","Data":"ea657f63575b49626dbe683db4b461926e65d695c5d11e105daec88cde15b106"}
Nov 02 09:46:56.139883 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.139856    4449 generic.go:356] "Generic (PLEG): container finished" podID="09ec8375-f77f-4b8f-a21c-9303968468ca" containerID="345b6f469a814342ef530b035b42b4dbf711cd303c3506d5bbd5ae894a0bcc1e" exitCode=0
Nov 02 09:46:56.140925 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.140905    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"
Nov 02 09:46:56.141237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.141036    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv" event={"ID":"09ec8375-f77f-4b8f-a21c-9303968468ca","Type":"ContainerDied","Data":"345b6f469a814342ef530b035b42b4dbf711cd303c3506d5bbd5ae894a0bcc1e"}
Nov 02 09:46:56.141237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.141077    4449 scope.go:122] "RemoveContainer" containerID="345b6f469a814342ef530b035b42b4dbf711cd303c3506d5bbd5ae894a0bcc1e"
Nov 02 09:46:56.300246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.298435    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"
Nov 02 09:46:56.315382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.315349    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ec8375-f77f-4b8f-a21c-9303968468ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ec8375-f77f-4b8f-a21c-9303968468ca-plugins-config-volume\") pod \"09ec8375-f77f-4b8f-a21c-9303968468ca\" (UID: \"09ec8375-f77f-4b8f-a21c-9303968468ca\") "
Nov 02 09:46:56.315752 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.315732    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ec8375-f77f-4b8f-a21c-9303968468ca-kube-api-access-bvb5z\" (UniqueName: \"kubernetes.io/projected/09ec8375-f77f-4b8f-a21c-9303968468ca-kube-api-access-bvb5z\") pod \"09ec8375-f77f-4b8f-a21c-9303968468ca\" (UID: \"09ec8375-f77f-4b8f-a21c-9303968468ca\") "
Nov 02 09:46:56.335472 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.317116    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ec8375-f77f-4b8f-a21c-9303968468ca-plugins-config-volume" pod "09ec8375-f77f-4b8f-a21c-9303968468ca" (UID: "09ec8375-f77f-4b8f-a21c-9303968468ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:46:56.335472 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.326916    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ec8375-f77f-4b8f-a21c-9303968468ca-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:56.335472 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.331801    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ec8375-f77f-4b8f-a21c-9303968468ca-kube-api-access-bvb5z" pod "09ec8375-f77f-4b8f-a21c-9303968468ca" (UID: "09ec8375-f77f-4b8f-a21c-9303968468ca"). InnerVolumeSpecName "kube-api-access-bvb5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:56.428201 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.428164    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f72acb2-73e1-4e88-8639-de0fb5ddce30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72acb2-73e1-4e88-8639-de0fb5ddce30-plugins-config-volume\") pod \"7f72acb2-73e1-4e88-8639-de0fb5ddce30\" (UID: \"7f72acb2-73e1-4e88-8639-de0fb5ddce30\") "
Nov 02 09:46:56.428401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.428245    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f72acb2-73e1-4e88-8639-de0fb5ddce30-kube-api-access-s4n6c\" (UniqueName: \"kubernetes.io/projected/7f72acb2-73e1-4e88-8639-de0fb5ddce30-kube-api-access-s4n6c\") pod \"7f72acb2-73e1-4e88-8639-de0fb5ddce30\" (UID: \"7f72acb2-73e1-4e88-8639-de0fb5ddce30\") "
Nov 02 09:46:56.428562 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.428548    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvb5z\" (UniqueName: \"kubernetes.io/projected/09ec8375-f77f-4b8f-a21c-9303968468ca-kube-api-access-bvb5z\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:56.428670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.428638    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f72acb2-73e1-4e88-8639-de0fb5ddce30-plugins-config-volume" pod "7f72acb2-73e1-4e88-8639-de0fb5ddce30" (UID: "7f72acb2-73e1-4e88-8639-de0fb5ddce30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:46:56.431611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.431579    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f72acb2-73e1-4e88-8639-de0fb5ddce30-kube-api-access-s4n6c" pod "7f72acb2-73e1-4e88-8639-de0fb5ddce30" (UID: "7f72acb2-73e1-4e88-8639-de0fb5ddce30"). InnerVolumeSpecName "kube-api-access-s4n6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:56.515798 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.515757    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"]
Nov 02 09:46:56.520143 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.520113    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d2e277-579fb69f6c-8plsv"]
Nov 02 09:46:56.530078 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.530049    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72acb2-73e1-4e88-8639-de0fb5ddce30-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:56.530078 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.530075    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4n6c\" (UniqueName: \"kubernetes.io/projected/7f72acb2-73e1-4e88-8639-de0fb5ddce30-kube-api-access-s4n6c\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:56.769306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.766664    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 09:46:56.769306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:56.766731    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" podUID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 09:46:57.016136 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.016053    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:46:57.017193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.016885    4449 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:46:57.017193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.016942    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk" podUID="7ef35a63-382b-4978-a07d-197c7775de78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 09:46:57.167243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.158706    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n" event={"ID":"7f72acb2-73e1-4e88-8639-de0fb5ddce30","Type":"ContainerDied","Data":"8cfb5caae6ee540726c9c5eab1c5e4bf5445a2931ef96863a45d51fcc1675815"}
Nov 02 09:46:57.167630 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.167606    4449 scope.go:122] "RemoveContainer" containerID="ea657f63575b49626dbe683db4b461926e65d695c5d11e105daec88cde15b106"
Nov 02 09:46:57.172640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.166335    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"
Nov 02 09:46:57.214566 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.214533    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ec8375-f77f-4b8f-a21c-9303968468ca" path="/var/lib/kubelet/pods/09ec8375-f77f-4b8f-a21c-9303968468ca/volumes"
Nov 02 09:46:57.234292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.234254    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"]
Nov 02 09:46:57.244256 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.244203    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d2e277-6bdc676c5f-g2r9n"]
Nov 02 09:46:57.787245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.777361    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 09:46:57.787245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.777450    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" podUID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 09:46:57.934329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.912287    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"]
Nov 02 09:46:57.934329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.914305    4449 cpu_manager.go:414] "containerMap: removing container" podUID="09ec8375-f77f-4b8f-a21c-9303968468ca" containerName="payload-processing"
Nov 02 09:46:57.934329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.914334    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ec8375-f77f-4b8f-a21c-9303968468ca" containerName="payload-processing"
Nov 02 09:46:57.934329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.914365    4449 cpu_manager.go:414] "containerMap: removing container" podUID="7f72acb2-73e1-4e88-8639-de0fb5ddce30" containerName="payload-pre-processing"
Nov 02 09:46:57.934329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.914373    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f72acb2-73e1-4e88-8639-de0fb5ddce30" containerName="payload-pre-processing"
Nov 02 09:46:57.934329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.917298    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ec8375-f77f-4b8f-a21c-9303968468ca" containerName="payload-processing"
Nov 02 09:46:57.934329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:57.917369    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f72acb2-73e1-4e88-8639-de0fb5ddce30" containerName="payload-pre-processing"
Nov 02 09:46:58.101423 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.100335    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"]
Nov 02 09:46:58.101423 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.100517    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"
Nov 02 09:46:58.105343 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.105303    4449 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:46:58.105530 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.105376    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk" podUID="7ef35a63-382b-4978-a07d-197c7775de78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 09:46:58.109467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.109438    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8743a5-dockercfg-zbwgb\""
Nov 02 09:46:58.111046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.111026    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8743a5\""
Nov 02 09:46:58.248858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.248818    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4024dd80-c4b0-4a24-a983-ba34eb385c57-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d\" (UID: \"4024dd80-c4b0-4a24-a983-ba34eb385c57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"
Nov 02 09:46:58.249040 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.248972    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9m7z\" (UniqueName: \"kubernetes.io/projected/4024dd80-c4b0-4a24-a983-ba34eb385c57-kube-api-access-c9m7z\") pod \"payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d\" (UID: \"4024dd80-c4b0-4a24-a983-ba34eb385c57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"
Nov 02 09:46:58.354307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.353974    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9m7z\" (UniqueName: \"kubernetes.io/projected/4024dd80-c4b0-4a24-a983-ba34eb385c57-kube-api-access-c9m7z\") pod \"payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d\" (UID: \"4024dd80-c4b0-4a24-a983-ba34eb385c57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"
Nov 02 09:46:58.354307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.354248    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4024dd80-c4b0-4a24-a983-ba34eb385c57-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d\" (UID: \"4024dd80-c4b0-4a24-a983-ba34eb385c57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"
Nov 02 09:46:58.375243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.355248    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4024dd80-c4b0-4a24-a983-ba34eb385c57-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d\" (UID: \"4024dd80-c4b0-4a24-a983-ba34eb385c57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"
Nov 02 09:46:58.396242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.395105    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9m7z\" (UniqueName: \"kubernetes.io/projected/4024dd80-c4b0-4a24-a983-ba34eb385c57-kube-api-access-c9m7z\") pod \"payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d\" (UID: \"4024dd80-c4b0-4a24-a983-ba34eb385c57\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"
Nov 02 09:46:58.425278 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.425237    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"
Nov 02 09:46:58.774300 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.773793    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 09:46:58.774300 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.773857    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" podUID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 09:46:58.807798 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:58.807762    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"]
Nov 02 09:46:59.013090 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:59.013052    4449 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:46:59.013298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:59.013126    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk" podUID="7ef35a63-382b-4978-a07d-197c7775de78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 09:46:59.219470 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:59.219437    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f72acb2-73e1-4e88-8639-de0fb5ddce30" path="/var/lib/kubelet/pods/7f72acb2-73e1-4e88-8639-de0fb5ddce30/volumes"
Nov 02 09:46:59.223196 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:59.223160    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d" event={"ID":"4024dd80-c4b0-4a24-a983-ba34eb385c57","Type":"ContainerStarted","Data":"5df023dce23c1f31fac6da011c97f09adcfc20dcb72fa5567a3994efcea4e436"}
Nov 02 09:46:59.223374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:59.223246    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"
Nov 02 09:46:59.223374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:59.223272    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d" event={"ID":"4024dd80-c4b0-4a24-a983-ba34eb385c57","Type":"ContainerStarted","Data":"7db96204689d1732123823c935a43bda06da337c094e7737e05bf05980371862"}
Nov 02 09:46:59.244345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:59.244283    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d" podStartSLOduration=2.244265252 podStartE2EDuration="2.244265252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:59.241654344 +0000 UTC m=+696.179941138" watchObservedRunningTime="2026-11-02 09:46:59.244265252 +0000 UTC m=+696.182552027"
Nov 02 09:46:59.765698 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:59.765650    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 09:46:59.765971 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:46:59.765938    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" podUID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 09:47:00.013182 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:00.013136    4449 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:47:00.013663 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:00.013197    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk" podUID="7ef35a63-382b-4978-a07d-197c7775de78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 09:47:00.772742 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:00.772700    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 09:47:00.773139 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:00.773106    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" podUID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 09:47:00.823442 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:00.821649    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:47:01.012683 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:01.012640    4449 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:47:01.012866 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:01.012712    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk" podUID="7ef35a63-382b-4978-a07d-197c7775de78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 09:47:01.910973 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:01.910925    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:47:01.912521 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:01.912499    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:47:01.914057 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:01.914031    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 09:47:02.024028 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:02.023998    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:47:02.025624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:02.025597    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:47:04.149174 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.149136    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"]
Nov 02 09:47:04.154727 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.154693    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"
Nov 02 09:47:04.159726 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.159325    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:47:04.160818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.160529    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:47:04.160818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.160637    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:47:04.160818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.160723    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:47:04.166722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.166449    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"]
Nov 02 09:47:04.284605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.284567    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa0054f1-bg5ng\" (UID: \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"
Nov 02 09:47:04.284791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.284623    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa0054f1-bg5ng\" (UID: \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"
Nov 02 09:47:04.284933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.284910    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69nkv\" (UniqueName: \"kubernetes.io/projected/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-kube-api-access-69nkv\") pod \"maas-api-revoke-sub-aa0054f1-bg5ng\" (UID: \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"
Nov 02 09:47:04.386407 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.386367    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa0054f1-bg5ng\" (UID: \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"
Nov 02 09:47:04.386606 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.386416    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa0054f1-bg5ng\" (UID: \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"
Nov 02 09:47:04.386606 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.386471    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69nkv\" (UniqueName: \"kubernetes.io/projected/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-kube-api-access-69nkv\") pod \"maas-api-revoke-sub-aa0054f1-bg5ng\" (UID: \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"
Nov 02 09:47:04.387145 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.387122    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa0054f1-bg5ng\" (UID: \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"
Nov 02 09:47:04.389026 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.389002    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa0054f1-bg5ng\" (UID: \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"
Nov 02 09:47:04.399267 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.399180    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69nkv\" (UniqueName: \"kubernetes.io/projected/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-kube-api-access-69nkv\") pod \"maas-api-revoke-sub-aa0054f1-bg5ng\" (UID: \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"
Nov 02 09:47:04.476356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.476324    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"
Nov 02 09:47:04.766242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.758903    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"]
Nov 02 09:47:04.766242 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:04.758949    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0452787b_5cbb_492a_9c7c_7e36a6b6e41c.slice/crio-653bae5acd0a2e8b73f4fef0ab1ca103a7a54baa2bb1d39ae7b450d4ec78eb61 WatchSource:0}: Error finding container 653bae5acd0a2e8b73f4fef0ab1ca103a7a54baa2bb1d39ae7b450d4ec78eb61: Status 404 returned error can't find the container with id 653bae5acd0a2e8b73f4fef0ab1ca103a7a54baa2bb1d39ae7b450d4ec78eb61
Nov 02 09:47:04.795012 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.794975    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7366-ae9450"]
Nov 02 09:47:04.803508 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.803480    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-ae9450"
Nov 02 09:47:04.806559 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.805801    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-ae9450"]
Nov 02 09:47:04.807919 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.807895    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8bqk\""
Nov 02 09:47:04.894403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.894360    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqjmx\" (UniqueName: \"kubernetes.io/projected/00ecbe25-9f71-49bb-bbc0-f0a6440f089d-kube-api-access-lqjmx\") pod \"test-curl-7366-ae9450\" (UID: \"00ecbe25-9f71-49bb-bbc0-f0a6440f089d\") " pod="opendatahub/test-curl-7366-ae9450"
Nov 02 09:47:04.995695 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:04.995658    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqjmx\" (UniqueName: \"kubernetes.io/projected/00ecbe25-9f71-49bb-bbc0-f0a6440f089d-kube-api-access-lqjmx\") pod \"test-curl-7366-ae9450\" (UID: \"00ecbe25-9f71-49bb-bbc0-f0a6440f089d\") " pod="opendatahub/test-curl-7366-ae9450"
Nov 02 09:47:05.008148 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:05.008062    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqjmx\" (UniqueName: \"kubernetes.io/projected/00ecbe25-9f71-49bb-bbc0-f0a6440f089d-kube-api-access-lqjmx\") pod \"test-curl-7366-ae9450\" (UID: \"00ecbe25-9f71-49bb-bbc0-f0a6440f089d\") " pod="opendatahub/test-curl-7366-ae9450"
Nov 02 09:47:05.120730 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:05.120687    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-ae9450"
Nov 02 09:47:05.400698 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:05.400653    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-ae9450"]
Nov 02 09:47:05.402562 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:05.402534    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00ecbe25_9f71_49bb_bbc0_f0a6440f089d.slice/crio-3fb30944226f15006e425b273ed7fec020f64a29230bb2a725bc43ecfb2840f3 WatchSource:0}: Error finding container 3fb30944226f15006e425b273ed7fec020f64a29230bb2a725bc43ecfb2840f3: Status 404 returned error can't find the container with id 3fb30944226f15006e425b273ed7fec020f64a29230bb2a725bc43ecfb2840f3
Nov 02 09:47:05.482680 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:05.482654    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng" event={"ID":"0452787b-5cbb-492a-9c7c-7e36a6b6e41c","Type":"ContainerStarted","Data":"dc2dc5f5c2b596787a2138089e39c739dd9d97c2bf2bc463a14629bb09b09774"}
Nov 02 09:47:05.482807 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:05.482704    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng" event={"ID":"0452787b-5cbb-492a-9c7c-7e36a6b6e41c","Type":"ContainerStarted","Data":"653bae5acd0a2e8b73f4fef0ab1ca103a7a54baa2bb1d39ae7b450d4ec78eb61"}
Nov 02 09:47:05.484281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:05.484259    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7366-ae9450" event={"ID":"00ecbe25-9f71-49bb-bbc0-f0a6440f089d","Type":"ContainerStarted","Data":"3fb30944226f15006e425b273ed7fec020f64a29230bb2a725bc43ecfb2840f3"}
Nov 02 09:47:05.500859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:05.500820    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng" podStartSLOduration=1.500806972 podStartE2EDuration="1.500806972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:05.499580406 +0000 UTC m=+702.437867181" watchObservedRunningTime="2026-11-02 09:47:05.500806972 +0000 UTC m=+702.439093747"
Nov 02 09:47:05.523780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:05.523721    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7366-ae9450" podStartSLOduration=1.523706875 podStartE2EDuration="1.523706875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:05.521014657 +0000 UTC m=+702.459301464" watchObservedRunningTime="2026-11-02 09:47:05.523706875 +0000 UTC m=+702.461993651"
Nov 02 09:47:06.237430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.237390    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7366-ae9450"]
Nov 02 09:47:06.242053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.242025    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7366-ae9450"]
Nov 02 09:47:06.494000 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.493923    4449 generic.go:356] "Generic (PLEG): container finished" podID="0452787b-5cbb-492a-9c7c-7e36a6b6e41c" containerID="dc2dc5f5c2b596787a2138089e39c739dd9d97c2bf2bc463a14629bb09b09774" exitCode=0
Nov 02 09:47:06.494386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.494000    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng" event={"ID":"0452787b-5cbb-492a-9c7c-7e36a6b6e41c","Type":"ContainerDied","Data":"dc2dc5f5c2b596787a2138089e39c739dd9d97c2bf2bc463a14629bb09b09774"}
Nov 02 09:47:06.547167 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:06.547128    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:47:06.814946 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.814917    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"]
Nov 02 09:47:06.815631 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.815607    4449 cpu_manager.go:414] "containerMap: removing container" podUID="00ecbe25-9f71-49bb-bbc0-f0a6440f089d" containerName="test-curl-7366-ae9450"
Nov 02 09:47:06.815631 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.815630    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ecbe25-9f71-49bb-bbc0-f0a6440f089d" containerName="test-curl-7366-ae9450"
Nov 02 09:47:06.815836 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.815821    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ecbe25-9f71-49bb-bbc0-f0a6440f089d" containerName="test-curl-7366-ae9450"
Nov 02 09:47:06.819436 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.819418    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"
Nov 02 09:47:06.825721 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.825699    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-629129-dockercfg-mkjn9\""
Nov 02 09:47:06.825859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.825849    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-629129\""
Nov 02 09:47:06.830720 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.830697    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"]
Nov 02 09:47:06.869118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.869093    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"]
Nov 02 09:47:06.873058 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.873038    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"
Nov 02 09:47:06.886718 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.886654    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"]
Nov 02 09:47:06.919178 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.919140    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrr4s\" (UniqueName: \"kubernetes.io/projected/d679fb2c-7057-4104-9b55-3cfa4fcde09c-kube-api-access-zrr4s\") pod \"payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv\" (UID: \"d679fb2c-7057-4104-9b55-3cfa4fcde09c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"
Nov 02 09:47:06.919178 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.919190    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czfwz\" (UniqueName: \"kubernetes.io/projected/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-kube-api-access-czfwz\") pod \"payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79\" (UID: \"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"
Nov 02 09:47:06.919462 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.919379    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79\" (UID: \"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"
Nov 02 09:47:06.919462 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:06.919425    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d679fb2c-7057-4104-9b55-3cfa4fcde09c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv\" (UID: \"d679fb2c-7057-4104-9b55-3cfa4fcde09c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"
Nov 02 09:47:07.020826 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.020782    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrr4s\" (UniqueName: \"kubernetes.io/projected/d679fb2c-7057-4104-9b55-3cfa4fcde09c-kube-api-access-zrr4s\") pod \"payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv\" (UID: \"d679fb2c-7057-4104-9b55-3cfa4fcde09c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"
Nov 02 09:47:07.021035 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.020840    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czfwz\" (UniqueName: \"kubernetes.io/projected/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-kube-api-access-czfwz\") pod \"payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79\" (UID: \"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"
Nov 02 09:47:07.021035 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.020893    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79\" (UID: \"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"
Nov 02 09:47:07.021035 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.020921    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d679fb2c-7057-4104-9b55-3cfa4fcde09c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv\" (UID: \"d679fb2c-7057-4104-9b55-3cfa4fcde09c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"
Nov 02 09:47:07.022086 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.022056    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d679fb2c-7057-4104-9b55-3cfa4fcde09c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv\" (UID: \"d679fb2c-7057-4104-9b55-3cfa4fcde09c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"
Nov 02 09:47:07.022624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.022605    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79\" (UID: \"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"
Nov 02 09:47:07.034449 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.034416    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czfwz\" (UniqueName: \"kubernetes.io/projected/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-kube-api-access-czfwz\") pod \"payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79\" (UID: \"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"
Nov 02 09:47:07.036182 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.036144    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrr4s\" (UniqueName: \"kubernetes.io/projected/d679fb2c-7057-4104-9b55-3cfa4fcde09c-kube-api-access-zrr4s\") pod \"payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv\" (UID: \"d679fb2c-7057-4104-9b55-3cfa4fcde09c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"
Nov 02 09:47:07.144924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.144840    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"
Nov 02 09:47:07.188106 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.188047    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"
Nov 02 09:47:07.427487 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:07.427457    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6bc134_cd0f_42b5_b77b_0f0eb9fc2b2f.slice/crio-59fe80233168362dbf2cfde147f8cba24a8d51e35a0e7599161c02e97e028b22 WatchSource:0}: Error finding container 59fe80233168362dbf2cfde147f8cba24a8d51e35a0e7599161c02e97e028b22: Status 404 returned error can't find the container with id 59fe80233168362dbf2cfde147f8cba24a8d51e35a0e7599161c02e97e028b22
Nov 02 09:47:07.427663 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.427640    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"]
Nov 02 09:47:07.486975 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.486944    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"]
Nov 02 09:47:07.492023 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.491998    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.496423 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.496388    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"]
Nov 02 09:47:07.514875 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.514841    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"]
Nov 02 09:47:07.528126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.528085    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79" event={"ID":"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f","Type":"ContainerStarted","Data":"59fe80233168362dbf2cfde147f8cba24a8d51e35a0e7599161c02e97e028b22"}
Nov 02 09:47:07.530479 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.530421    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7366-ae9450" podUID="00ecbe25-9f71-49bb-bbc0-f0a6440f089d" containerName="test-curl-7366-ae9450" containerID="cri-o://c603d80df947e0c56e343407270ab96e630521d0ee9ef3ff1608af87fcd0fcde" gracePeriod=2
Nov 02 09:47:07.530971 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.530941    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv" event={"ID":"d679fb2c-7057-4104-9b55-3cfa4fcde09c","Type":"ContainerStarted","Data":"4800932682cc70f023add2114c16cc689a82d399c0bcfff65a1392e4b09fc3e1"}
Nov 02 09:47:07.531817 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.531683    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"
Nov 02 09:47:07.629024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.628988    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.629171 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.629135    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.629262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.629168    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqr4k\" (UniqueName: \"kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-kube-api-access-nqr4k\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.629421 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.629372    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.629909 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.629883    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.630001 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.629948    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.630001 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.629978    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.630162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.630064    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.630162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.630156    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.687177 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.687152    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"
Nov 02 09:47:07.717111 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.715981    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79" podStartSLOduration=1.715959667 podStartE2EDuration="1.715959667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:07.549832587 +0000 UTC m=+704.488119376" watchObservedRunningTime="2026-11-02 09:47:07.715959667 +0000 UTC m=+704.654246444"
Nov 02 09:47:07.731796 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.731754    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-xks-ca\") pod \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\" (UID: \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\") "
Nov 02 09:47:07.731982 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.731875    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-kube-api-access-69nkv\" (UniqueName: \"kubernetes.io/projected/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-kube-api-access-69nkv\") pod \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\" (UID: \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\") "
Nov 02 09:47:07.732065 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.732050    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-openshift-ca\") pod \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\" (UID: \"0452787b-5cbb-492a-9c7c-7e36a6b6e41c\") "
Nov 02 09:47:07.732277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.732259    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.732380 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.732362    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.732440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.732424    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.732514 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.732497    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.732572 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.732534    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqr4k\" (UniqueName: \"kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-kube-api-access-nqr4k\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.732572 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.732564    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.732684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.732668    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.732739 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.732728    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.732792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.732760    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.733637 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.733613    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.733862 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.733831    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.733945 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.733868    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.734562 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.734355    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.734562 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.734499    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-openshift-ca" pod "0452787b-5cbb-492a-9c7c-7e36a6b6e41c" (UID: "0452787b-5cbb-492a-9c7c-7e36a6b6e41c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:47:07.737944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.737898    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.738640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.738614    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.741907 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.741870    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-xks-ca" pod "0452787b-5cbb-492a-9c7c-7e36a6b6e41c" (UID: "0452787b-5cbb-492a-9c7c-7e36a6b6e41c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:47:07.746491 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.746464    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.746940 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.746906    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.748253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.748203    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-kube-api-access-69nkv" pod "0452787b-5cbb-492a-9c7c-7e36a6b6e41c" (UID: "0452787b-5cbb-492a-9c7c-7e36a6b6e41c"). InnerVolumeSpecName "kube-api-access-69nkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:07.748859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.748839    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqr4k\" (UniqueName: \"kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-kube-api-access-nqr4k\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.828041 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.828010    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:07.833981 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.833954    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:07.834053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.833991    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:07.834053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:07.834009    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69nkv\" (UniqueName: \"kubernetes.io/projected/0452787b-5cbb-492a-9c7c-7e36a6b6e41c-kube-api-access-69nkv\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:08.113649 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.113618    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"]
Nov 02 09:47:08.114334 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:08.114309    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7b8d80_80a7_47b3_876d_dce3ebb293fa.slice/crio-13f67a8976035656d4116ca078e5e40a8ae3c5b6de1e0eaa9f3a80ec9e6e30ff WatchSource:0}: Error finding container 13f67a8976035656d4116ca078e5e40a8ae3c5b6de1e0eaa9f3a80ec9e6e30ff: Status 404 returned error can't find the container with id 13f67a8976035656d4116ca078e5e40a8ae3c5b6de1e0eaa9f3a80ec9e6e30ff
Nov 02 09:47:08.117316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.117283    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:47:08.117398 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.117355    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:47:08.117398 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.117387    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:47:08.538310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.538281    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"
Nov 02 09:47:08.538972 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.538288    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng" event={"ID":"0452787b-5cbb-492a-9c7c-7e36a6b6e41c","Type":"ContainerDied","Data":"653bae5acd0a2e8b73f4fef0ab1ca103a7a54baa2bb1d39ae7b450d4ec78eb61"}
Nov 02 09:47:08.538972 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.538409    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="653bae5acd0a2e8b73f4fef0ab1ca103a7a54baa2bb1d39ae7b450d4ec78eb61"
Nov 02 09:47:08.541675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.541647    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" event={"ID":"0c7b8d80-80a7-47b3-876d-dce3ebb293fa","Type":"ContainerStarted","Data":"76921ab24f098a1f4d6601045e75474b37ddc61699dddc97b7a409fcd2825720"}
Nov 02 09:47:08.541817 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.541704    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" event={"ID":"0c7b8d80-80a7-47b3-876d-dce3ebb293fa","Type":"ContainerStarted","Data":"13f67a8976035656d4116ca078e5e40a8ae3c5b6de1e0eaa9f3a80ec9e6e30ff"}
Nov 02 09:47:08.543469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.543439    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv" event={"ID":"d679fb2c-7057-4104-9b55-3cfa4fcde09c","Type":"ContainerStarted","Data":"1040990dd13ddc3f638a1ba13e02434d80f61c72d61f006e7745505cf84469d6"}
Nov 02 09:47:08.543577 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.543530    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"
Nov 02 09:47:08.545071 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.545046    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79" event={"ID":"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f","Type":"ContainerStarted","Data":"308bb8e5c0caa410f11cf281122f6d5722f82285c5ccaacb3fdb877c7d707fb5"}
Nov 02 09:47:08.568922 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.568879    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" podStartSLOduration=1.568863752 podStartE2EDuration="1.568863752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:08.565772313 +0000 UTC m=+705.504059110" watchObservedRunningTime="2026-11-02 09:47:08.568863752 +0000 UTC m=+705.507150725"
Nov 02 09:47:08.590355 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.590305    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv" podStartSLOduration=2.5902913180000002 podStartE2EDuration="2.590291318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:08.589276685 +0000 UTC m=+705.527563492" watchObservedRunningTime="2026-11-02 09:47:08.590291318 +0000 UTC m=+705.528578094"
Nov 02 09:47:08.828642 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:08.828556    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:09.831764 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:09.830959    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 09:47:09.831764 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:09.831029    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 09:47:09.987528 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:09.986689    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-ae9450"
Nov 02 09:47:10.098704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:10.098617    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ecbe25-9f71-49bb-bbc0-f0a6440f089d-kube-api-access-lqjmx\" (UniqueName: \"kubernetes.io/projected/00ecbe25-9f71-49bb-bbc0-f0a6440f089d-kube-api-access-lqjmx\") pod \"00ecbe25-9f71-49bb-bbc0-f0a6440f089d\" (UID: \"00ecbe25-9f71-49bb-bbc0-f0a6440f089d\") "
Nov 02 09:47:10.101265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:10.101194    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ecbe25-9f71-49bb-bbc0-f0a6440f089d-kube-api-access-lqjmx" pod "00ecbe25-9f71-49bb-bbc0-f0a6440f089d" (UID: "00ecbe25-9f71-49bb-bbc0-f0a6440f089d"). InnerVolumeSpecName "kube-api-access-lqjmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:10.200159 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:10.200123    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqjmx\" (UniqueName: \"kubernetes.io/projected/00ecbe25-9f71-49bb-bbc0-f0a6440f089d-kube-api-access-lqjmx\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:10.232537 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:10.232505    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"
Nov 02 09:47:10.598723 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:10.598679    4449 generic.go:356] "Generic (PLEG): container finished" podID="00ecbe25-9f71-49bb-bbc0-f0a6440f089d" containerID="c603d80df947e0c56e343407270ab96e630521d0ee9ef3ff1608af87fcd0fcde" exitCode=137
Nov 02 09:47:10.599232 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:10.599196    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-ae9450"
Nov 02 09:47:10.599386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:10.599372    4449 scope.go:122] "RemoveContainer" containerID="c603d80df947e0c56e343407270ab96e630521d0ee9ef3ff1608af87fcd0fcde"
Nov 02 09:47:10.631270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:10.629393    4449 scope.go:122] "RemoveContainer" containerID="c603d80df947e0c56e343407270ab96e630521d0ee9ef3ff1608af87fcd0fcde"
Nov 02 09:47:10.631270 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:10.631138    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c603d80df947e0c56e343407270ab96e630521d0ee9ef3ff1608af87fcd0fcde\": container with ID starting with c603d80df947e0c56e343407270ab96e630521d0ee9ef3ff1608af87fcd0fcde not found: ID does not exist" containerID="c603d80df947e0c56e343407270ab96e630521d0ee9ef3ff1608af87fcd0fcde"
Nov 02 09:47:10.631270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:10.631174    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c603d80df947e0c56e343407270ab96e630521d0ee9ef3ff1608af87fcd0fcde"} err="failed to get container status \"c603d80df947e0c56e343407270ab96e630521d0ee9ef3ff1608af87fcd0fcde\": rpc error: code = NotFound desc = could not find container \"c603d80df947e0c56e343407270ab96e630521d0ee9ef3ff1608af87fcd0fcde\": container with ID starting with c603d80df947e0c56e343407270ab96e630521d0ee9ef3ff1608af87fcd0fcde not found: ID does not exist"
Nov 02 09:47:10.671537 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:10.671434    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-ae9450\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="00ecbe25-9f71-49bb-bbc0-f0a6440f089d" pod="opendatahub/test-curl-7366-ae9450"
Nov 02 09:47:10.828811 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:10.828422    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 09:47:10.828811 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:10.828493    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 09:47:11.216170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:11.214595    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ecbe25-9f71-49bb-bbc0-f0a6440f089d" path="/var/lib/kubelet/pods/00ecbe25-9f71-49bb-bbc0-f0a6440f089d/volumes"
Nov 02 09:47:12.070249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:12.069437    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 09:47:12.070249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:12.069502    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 09:47:12.846241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:12.845406    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 09:47:12.846241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:12.845469    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 09:47:13.829077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:13.829047    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 09:47:13.829340 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:13.829314    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 09:47:14.401852 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.401799    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"]
Nov 02 09:47:14.402808 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.402783    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0452787b-5cbb-492a-9c7c-7e36a6b6e41c" containerName="revoke-keys"
Nov 02 09:47:14.402922 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.402809    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0452787b-5cbb-492a-9c7c-7e36a6b6e41c" containerName="revoke-keys"
Nov 02 09:47:14.403000 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.402979    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0452787b-5cbb-492a-9c7c-7e36a6b6e41c" containerName="revoke-keys"
Nov 02 09:47:14.408240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.408056    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"
Nov 02 09:47:14.411995 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.411965    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"]
Nov 02 09:47:14.412725 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.412412    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:47:14.413288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.413254    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:47:14.413393 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.413305    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:47:14.413393 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.413336    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:47:14.507096 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.507046    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70a395c3-pxhwb\" (UID: \"a2de4e9a-92da-4f06-a101-94af13ff5099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"
Nov 02 09:47:14.507366 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.507154    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70a395c3-pxhwb\" (UID: \"a2de4e9a-92da-4f06-a101-94af13ff5099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"
Nov 02 09:47:14.507366 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.507197    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tkqz\" (UniqueName: \"kubernetes.io/projected/a2de4e9a-92da-4f06-a101-94af13ff5099-kube-api-access-5tkqz\") pod \"maas-api-revoke-sub-70a395c3-pxhwb\" (UID: \"a2de4e9a-92da-4f06-a101-94af13ff5099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"
Nov 02 09:47:14.608286 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.608231    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70a395c3-pxhwb\" (UID: \"a2de4e9a-92da-4f06-a101-94af13ff5099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"
Nov 02 09:47:14.608524 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.608370    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70a395c3-pxhwb\" (UID: \"a2de4e9a-92da-4f06-a101-94af13ff5099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"
Nov 02 09:47:14.608524 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.608419    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tkqz\" (UniqueName: \"kubernetes.io/projected/a2de4e9a-92da-4f06-a101-94af13ff5099-kube-api-access-5tkqz\") pod \"maas-api-revoke-sub-70a395c3-pxhwb\" (UID: \"a2de4e9a-92da-4f06-a101-94af13ff5099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"
Nov 02 09:47:14.609084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.609061    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70a395c3-pxhwb\" (UID: \"a2de4e9a-92da-4f06-a101-94af13ff5099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"
Nov 02 09:47:14.613832 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.613792    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70a395c3-pxhwb\" (UID: \"a2de4e9a-92da-4f06-a101-94af13ff5099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"
Nov 02 09:47:14.621328 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.621295    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tkqz\" (UniqueName: \"kubernetes.io/projected/a2de4e9a-92da-4f06-a101-94af13ff5099-kube-api-access-5tkqz\") pod \"maas-api-revoke-sub-70a395c3-pxhwb\" (UID: \"a2de4e9a-92da-4f06-a101-94af13ff5099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"
Nov 02 09:47:14.724760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.723928    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"
Nov 02 09:47:14.871257 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.870140    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 09:47:14.871257 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:14.870198    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 09:47:15.084382 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:15.084329    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2de4e9a_92da_4f06_a101_94af13ff5099.slice/crio-36b13d8b81f7e383e62a7c5e21255aff6660602c51e83d533aace11aa587d22f WatchSource:0}: Error finding container 36b13d8b81f7e383e62a7c5e21255aff6660602c51e83d533aace11aa587d22f: Status 404 returned error can't find the container with id 36b13d8b81f7e383e62a7c5e21255aff6660602c51e83d533aace11aa587d22f
Nov 02 09:47:15.084634 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:15.084609    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"]
Nov 02 09:47:15.126761 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:15.126718    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb" event={"ID":"a2de4e9a-92da-4f06-a101-94af13ff5099","Type":"ContainerStarted","Data":"36b13d8b81f7e383e62a7c5e21255aff6660602c51e83d533aace11aa587d22f"}
Nov 02 09:47:15.828594 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:15.828557    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 09:47:15.829046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:15.828626    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 09:47:16.140370 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.140333    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb" event={"ID":"a2de4e9a-92da-4f06-a101-94af13ff5099","Type":"ContainerStarted","Data":"414f135912906b4615f7a3ba3705cda95f844fc4743912f3f3c273e759d0bb72"}
Nov 02 09:47:16.193608 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.193541    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb" podStartSLOduration=2.193518673 podStartE2EDuration="2.193518673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:16.160730043 +0000 UTC m=+713.099016837" watchObservedRunningTime="2026-11-02 09:47:16.193518673 +0000 UTC m=+713.131805450"
Nov 02 09:47:16.680320 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.680274    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"]
Nov 02 09:47:16.685883 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.685857    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"
Nov 02 09:47:16.693840 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.693799    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"]
Nov 02 09:47:16.743460 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.743412    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74w7\" (UniqueName: \"kubernetes.io/projected/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-kube-api-access-l74w7\") pod \"maas-api-revoke-sub-b714b53e-ks7bf\" (UID: \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"
Nov 02 09:47:16.743679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.743479    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b714b53e-ks7bf\" (UID: \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"
Nov 02 09:47:16.743759 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.743706    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b714b53e-ks7bf\" (UID: \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"
Nov 02 09:47:16.845302 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.845268    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b714b53e-ks7bf\" (UID: \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"
Nov 02 09:47:16.845772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.845388    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74w7\" (UniqueName: \"kubernetes.io/projected/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-kube-api-access-l74w7\") pod \"maas-api-revoke-sub-b714b53e-ks7bf\" (UID: \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"
Nov 02 09:47:16.845772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.845424    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b714b53e-ks7bf\" (UID: \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"
Nov 02 09:47:16.846078 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.846053    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b714b53e-ks7bf\" (UID: \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"
Nov 02 09:47:16.848095 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.848068    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b714b53e-ks7bf\" (UID: \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"
Nov 02 09:47:16.861962 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:16.861897    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74w7\" (UniqueName: \"kubernetes.io/projected/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-kube-api-access-l74w7\") pod \"maas-api-revoke-sub-b714b53e-ks7bf\" (UID: \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"
Nov 02 09:47:17.013871 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:17.013427    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"
Nov 02 09:47:17.186274 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:17.186231    4449 generic.go:356] "Generic (PLEG): container finished" podID="a2de4e9a-92da-4f06-a101-94af13ff5099" containerID="414f135912906b4615f7a3ba3705cda95f844fc4743912f3f3c273e759d0bb72" exitCode=0
Nov 02 09:47:17.186437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:17.186417    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb" event={"ID":"a2de4e9a-92da-4f06-a101-94af13ff5099","Type":"ContainerDied","Data":"414f135912906b4615f7a3ba3705cda95f844fc4743912f3f3c273e759d0bb72"}
Nov 02 09:47:17.378246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:17.377946    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"]
Nov 02 09:47:17.384525 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:17.384499    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a40410d_eea1_4a9e_a9de_0be97cf4bf1b.slice/crio-c9240b1b620e0e9a4f8fc629c839983aa80fc5de855e7c872ed485e71b56d449 WatchSource:0}: Error finding container c9240b1b620e0e9a4f8fc629c839983aa80fc5de855e7c872ed485e71b56d449: Status 404 returned error can't find the container with id c9240b1b620e0e9a4f8fc629c839983aa80fc5de855e7c872ed485e71b56d449
Nov 02 09:47:17.871363 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:17.871335    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:47:17.871815 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:17.871386    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:18.211210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.211164    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf" event={"ID":"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b","Type":"ContainerStarted","Data":"d3079b62c3566433e4cd5571c379e8a366d262bfa7cf1c48930331db9f1366a1"}
Nov 02 09:47:18.211501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.211477    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf" event={"ID":"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b","Type":"ContainerStarted","Data":"c9240b1b620e0e9a4f8fc629c839983aa80fc5de855e7c872ed485e71b56d449"}
Nov 02 09:47:18.231486 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.231425    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf" podStartSLOduration=2.231406207 podStartE2EDuration="2.231406207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:18.22977895 +0000 UTC m=+715.168065747" watchObservedRunningTime="2026-11-02 09:47:18.231406207 +0000 UTC m=+715.169692984"
Nov 02 09:47:18.423193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.423167    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"
Nov 02 09:47:18.600739 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.600698    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2de4e9a-92da-4f06-a101-94af13ff5099-kube-api-access-5tkqz\" (UniqueName: \"kubernetes.io/projected/a2de4e9a-92da-4f06-a101-94af13ff5099-kube-api-access-5tkqz\") pod \"a2de4e9a-92da-4f06-a101-94af13ff5099\" (UID: \"a2de4e9a-92da-4f06-a101-94af13ff5099\") "
Nov 02 09:47:18.600876 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.600817    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-openshift-ca\") pod \"a2de4e9a-92da-4f06-a101-94af13ff5099\" (UID: \"a2de4e9a-92da-4f06-a101-94af13ff5099\") "
Nov 02 09:47:18.600919 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.600884    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-xks-ca\") pod \"a2de4e9a-92da-4f06-a101-94af13ff5099\" (UID: \"a2de4e9a-92da-4f06-a101-94af13ff5099\") "
Nov 02 09:47:18.601286 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.601251    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-openshift-ca" pod "a2de4e9a-92da-4f06-a101-94af13ff5099" (UID: "a2de4e9a-92da-4f06-a101-94af13ff5099"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:47:18.601517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.601496    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:18.603112 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.603081    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-xks-ca" pod "a2de4e9a-92da-4f06-a101-94af13ff5099" (UID: "a2de4e9a-92da-4f06-a101-94af13ff5099"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:47:18.603589 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.603567    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2de4e9a-92da-4f06-a101-94af13ff5099-kube-api-access-5tkqz" pod "a2de4e9a-92da-4f06-a101-94af13ff5099" (UID: "a2de4e9a-92da-4f06-a101-94af13ff5099"). InnerVolumeSpecName "kube-api-access-5tkqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:18.703679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.703625    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2de4e9a-92da-4f06-a101-94af13ff5099-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:18.703679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.703676    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tkqz\" (UniqueName: \"kubernetes.io/projected/a2de4e9a-92da-4f06-a101-94af13ff5099-kube-api-access-5tkqz\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:18.714430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.714398    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:18.714870 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.714849    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:18.715834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:18.715815    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 09:47:19.217450 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:19.217408    4449 generic.go:356] "Generic (PLEG): container finished" podID="8a40410d-eea1-4a9e-a9de-0be97cf4bf1b" containerID="d3079b62c3566433e4cd5571c379e8a366d262bfa7cf1c48930331db9f1366a1" exitCode=0
Nov 02 09:47:19.217923 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:19.217523    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf" event={"ID":"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b","Type":"ContainerDied","Data":"d3079b62c3566433e4cd5571c379e8a366d262bfa7cf1c48930331db9f1366a1"}
Nov 02 09:47:19.222124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:19.222099    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"
Nov 02 09:47:19.222321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:19.222295    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb" event={"ID":"a2de4e9a-92da-4f06-a101-94af13ff5099","Type":"ContainerDied","Data":"36b13d8b81f7e383e62a7c5e21255aff6660602c51e83d533aace11aa587d22f"}
Nov 02 09:47:19.222432 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:19.222345    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36b13d8b81f7e383e62a7c5e21255aff6660602c51e83d533aace11aa587d22f"
Nov 02 09:47:19.592238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:19.591858    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"
Nov 02 09:47:19.592238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:19.591949    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"
Nov 02 09:47:20.563747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:20.563704    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"
Nov 02 09:47:20.733730 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:20.733677    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-openshift-ca\") pod \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\" (UID: \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\") "
Nov 02 09:47:20.733914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:20.733799    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-xks-ca\") pod \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\" (UID: \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\") "
Nov 02 09:47:20.733914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:20.733866    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-kube-api-access-l74w7\" (UniqueName: \"kubernetes.io/projected/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-kube-api-access-l74w7\") pod \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\" (UID: \"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b\") "
Nov 02 09:47:20.734867 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:20.734829    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-openshift-ca" pod "8a40410d-eea1-4a9e-a9de-0be97cf4bf1b" (UID: "8a40410d-eea1-4a9e-a9de-0be97cf4bf1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:47:20.736810 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:20.736764    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-xks-ca" pod "8a40410d-eea1-4a9e-a9de-0be97cf4bf1b" (UID: "8a40410d-eea1-4a9e-a9de-0be97cf4bf1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:47:20.740446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:20.739660    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-kube-api-access-l74w7" pod "8a40410d-eea1-4a9e-a9de-0be97cf4bf1b" (UID: "8a40410d-eea1-4a9e-a9de-0be97cf4bf1b"). InnerVolumeSpecName "kube-api-access-l74w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:20.835549 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:20.835509    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:20.835549 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:20.835546    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:20.835846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:20.835561    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l74w7\" (UniqueName: \"kubernetes.io/projected/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b-kube-api-access-l74w7\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:21.266313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:21.256131    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf" event={"ID":"8a40410d-eea1-4a9e-a9de-0be97cf4bf1b","Type":"ContainerDied","Data":"c9240b1b620e0e9a4f8fc629c839983aa80fc5de855e7c872ed485e71b56d449"}
Nov 02 09:47:21.266313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:21.256195    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9240b1b620e0e9a4f8fc629c839983aa80fc5de855e7c872ed485e71b56d449"
Nov 02 09:47:21.266313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:21.256291    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"
Nov 02 09:47:23.126589 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.126552    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"]
Nov 02 09:47:23.127621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.127594    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a2de4e9a-92da-4f06-a101-94af13ff5099" containerName="revoke-keys"
Nov 02 09:47:23.127621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.127621    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2de4e9a-92da-4f06-a101-94af13ff5099" containerName="revoke-keys"
Nov 02 09:47:23.127827 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.127660    4449 cpu_manager.go:414] "containerMap: removing container" podUID="8a40410d-eea1-4a9e-a9de-0be97cf4bf1b" containerName="revoke-keys"
Nov 02 09:47:23.127827 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.127668    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a40410d-eea1-4a9e-a9de-0be97cf4bf1b" containerName="revoke-keys"
Nov 02 09:47:23.127827 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.127807    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2de4e9a-92da-4f06-a101-94af13ff5099" containerName="revoke-keys"
Nov 02 09:47:23.127827 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.127821    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a40410d-eea1-4a9e-a9de-0be97cf4bf1b" containerName="revoke-keys"
Nov 02 09:47:23.130258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.130209    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.134829 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.134802    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-301be8\"/\"kube-root-ca.crt\""
Nov 02 09:47:23.137463 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.137439    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"]
Nov 02 09:47:23.138688 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.138665    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-301be8\"/\"default-dockercfg-z9kv9\""
Nov 02 09:47:23.138805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.138678    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-301be8\"/\"test-model-301be8-kserve-self-signed-certs\""
Nov 02 09:47:23.237346 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.237317    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-301be8\"/\"openshift-service-ca.crt\""
Nov 02 09:47:23.237741 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.237718    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-dshm\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.237786 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.237766    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-home\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.237837 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.237822    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tls-certs\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.237938 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.237922    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-model-cache\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.237992 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.237980    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tmp-dir\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.238053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.238038    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n895m\" (UniqueName: \"kubernetes.io/projected/5ed165dd-1500-4fd4-a5ae-5a35d894726e-kube-api-access-n895m\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.348822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.341697    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n895m\" (UniqueName: \"kubernetes.io/projected/5ed165dd-1500-4fd4-a5ae-5a35d894726e-kube-api-access-n895m\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.348822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.341818    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-dshm\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.348822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.341865    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-home\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.348822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.341887    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tls-certs\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.348822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.341942    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-model-cache\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.348822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.342005    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tmp-dir\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.348822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.346428    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-home\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.351596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.349956    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-model-cache\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.357885 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.354523    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-301be8\"/\"test-model-301be8-kserve-self-signed-certs\""
Nov 02 09:47:23.357885 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.355045    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tmp-dir\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.363352 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.363321    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-301be8\"/\"kube-root-ca.crt\""
Nov 02 09:47:23.364055 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.364024    4449 generic.go:356] "Generic (PLEG): container finished" podID="5928113f-5ce3-4403-b390-d52a356939af" containerID="d962a00c6aa93312933d4291fdbf5c290952111d56e3b4657c157b35e7af97c7" exitCode=0
Nov 02 09:47:23.364184 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.364085    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" event={"ID":"5928113f-5ce3-4403-b390-d52a356939af","Type":"ContainerDied","Data":"d962a00c6aa93312933d4291fdbf5c290952111d56e3b4657c157b35e7af97c7"}
Nov 02 09:47:23.364636 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.364617    4449 scope.go:122] "RemoveContainer" containerID="d962a00c6aa93312933d4291fdbf5c290952111d56e3b4657c157b35e7af97c7"
Nov 02 09:47:23.369843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.366748    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:47:23.369843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.366843    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:47:23.369843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.366919    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:47:23.371844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.371820    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-dshm\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.372688 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.372663    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tls-certs\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.373505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.373484    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n895m\" (UniqueName: \"kubernetes.io/projected/5ed165dd-1500-4fd4-a5ae-5a35d894726e-kube-api-access-n895m\") pod \"test-model-301be8-kserve-5955c9fdfc-qfwkz\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") " pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.448121 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.448099    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-301be8\"/\"default-dockercfg-z9kv9\""
Nov 02 09:47:23.453951 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.453929    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:23.743241 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:23.743198    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ed165dd_1500_4fd4_a5ae_5a35d894726e.slice/crio-0c380ac82e4677c5630bb9aed8688a4bb0c3ef53d917b56a260362706c342dec WatchSource:0}: Error finding container 0c380ac82e4677c5630bb9aed8688a4bb0c3ef53d917b56a260362706c342dec: Status 404 returned error can't find the container with id 0c380ac82e4677c5630bb9aed8688a4bb0c3ef53d917b56a260362706c342dec
Nov 02 09:47:23.743383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:23.743333    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"]
Nov 02 09:47:24.380555 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:24.380517    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz" event={"ID":"5ed165dd-1500-4fd4-a5ae-5a35d894726e","Type":"ContainerStarted","Data":"11822a424089808f6099254a352c407abd819c54e1d747af6d0a87835ebd1ec0"}
Nov 02 09:47:24.380981 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:24.380566    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz" event={"ID":"5ed165dd-1500-4fd4-a5ae-5a35d894726e","Type":"ContainerStarted","Data":"0c380ac82e4677c5630bb9aed8688a4bb0c3ef53d917b56a260362706c342dec"}
Nov 02 09:47:24.382841 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:24.382810    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" event={"ID":"5928113f-5ce3-4403-b390-d52a356939af","Type":"ContainerStarted","Data":"67d2ecf39dfa86c0ad1e4a3f5beff9205d0ffe4d3685994c76dd4c045b0f88d0"}
Nov 02 09:47:24.404694 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:24.404642    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz" podStartSLOduration=1.404628771 podStartE2EDuration="1.404628771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:24.401931743 +0000 UTC m=+721.340218523" watchObservedRunningTime="2026-11-02 09:47:24.404628771 +0000 UTC m=+721.342915546"
Nov 02 09:47:24.474415 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:24.474357    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:24.474633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:24.474458    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:47:24.476191 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:24.476163    4449 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 02 09:47:24.476325 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:24.476237    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" podUID="5928113f-5ce3-4403-b390-d52a356939af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 02 09:47:25.106326 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:25.106294    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:47:25.468755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:25.468655    4449 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 02 09:47:25.471137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:25.471102    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" podUID="5928113f-5ce3-4403-b390-d52a356939af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 02 09:47:26.459246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:26.455655    4449 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 02 09:47:26.459246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:26.455721    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" podUID="5928113f-5ce3-4403-b390-d52a356939af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 02 09:47:27.455665 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:27.455626    4449 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 02 09:47:27.456326 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:27.456291    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" podUID="5928113f-5ce3-4403-b390-d52a356939af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 02 09:47:28.463253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:28.457817    4449 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 02 09:47:28.463253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:28.457879    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" podUID="5928113f-5ce3-4403-b390-d52a356939af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 02 09:47:29.304928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.304887    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"]
Nov 02 09:47:29.310058 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.310027    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"
Nov 02 09:47:29.322040 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.322009    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:47:29.322643 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.322058    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:47:29.323811 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.323370    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:47:29.323811 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.323406    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:47:29.333977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.333936    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"]
Nov 02 09:47:29.467243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.467190    4449 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 02 09:47:29.467788 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.467746    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" podUID="5928113f-5ce3-4403-b390-d52a356939af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 02 09:47:29.476792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.476751    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a42c0e35-x2c6f\" (UID: \"41246da8-0317-4f59-a90e-6b4f02b28115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"
Nov 02 09:47:29.476994 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.476971    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx7fk\" (UniqueName: \"kubernetes.io/projected/41246da8-0317-4f59-a90e-6b4f02b28115-kube-api-access-wx7fk\") pod \"maas-api-revoke-sub-a42c0e35-x2c6f\" (UID: \"41246da8-0317-4f59-a90e-6b4f02b28115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"
Nov 02 09:47:29.477116 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.477093    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a42c0e35-x2c6f\" (UID: \"41246da8-0317-4f59-a90e-6b4f02b28115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"
Nov 02 09:47:29.578805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.578706    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7fk\" (UniqueName: \"kubernetes.io/projected/41246da8-0317-4f59-a90e-6b4f02b28115-kube-api-access-wx7fk\") pod \"maas-api-revoke-sub-a42c0e35-x2c6f\" (UID: \"41246da8-0317-4f59-a90e-6b4f02b28115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"
Nov 02 09:47:29.578805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.578782    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a42c0e35-x2c6f\" (UID: \"41246da8-0317-4f59-a90e-6b4f02b28115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"
Nov 02 09:47:29.579042 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.578875    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a42c0e35-x2c6f\" (UID: \"41246da8-0317-4f59-a90e-6b4f02b28115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"
Nov 02 09:47:29.579718 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.579686    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a42c0e35-x2c6f\" (UID: \"41246da8-0317-4f59-a90e-6b4f02b28115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"
Nov 02 09:47:29.581634 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.581599    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a42c0e35-x2c6f\" (UID: \"41246da8-0317-4f59-a90e-6b4f02b28115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"
Nov 02 09:47:29.597998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.597966    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx7fk\" (UniqueName: \"kubernetes.io/projected/41246da8-0317-4f59-a90e-6b4f02b28115-kube-api-access-wx7fk\") pod \"maas-api-revoke-sub-a42c0e35-x2c6f\" (UID: \"41246da8-0317-4f59-a90e-6b4f02b28115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"
Nov 02 09:47:29.628746 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.628704    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"
Nov 02 09:47:29.914539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:29.914507    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"]
Nov 02 09:47:29.915424 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:29.915396    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41246da8_0317_4f59_a90e_6b4f02b28115.slice/crio-e9bc9e71c3bcaebd97f6a962d9218063a0fe49132620a3db4a16657d10aa69f5 WatchSource:0}: Error finding container e9bc9e71c3bcaebd97f6a962d9218063a0fe49132620a3db4a16657d10aa69f5: Status 404 returned error can't find the container with id e9bc9e71c3bcaebd97f6a962d9218063a0fe49132620a3db4a16657d10aa69f5
Nov 02 09:47:30.378661 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.378616    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"]
Nov 02 09:47:30.384103 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.384066    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.392148 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.392123    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8743a5\"/\"kube-root-ca.crt\""
Nov 02 09:47:30.392148 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.392132    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8743a5\"/\"openshift-service-ca.crt\""
Nov 02 09:47:30.392847 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.392799    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8743a5\"/\"default-dockercfg-sbj8d\""
Nov 02 09:47:30.392847 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.392831    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8743a5\"/\"test-model-8743a5-kserve-self-signed-certs\""
Nov 02 09:47:30.395983 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.395960    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"]
Nov 02 09:47:30.434059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.433958    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f" event={"ID":"41246da8-0317-4f59-a90e-6b4f02b28115","Type":"ContainerStarted","Data":"3c6d78bda774cdc613412a3f52f9d3d7973758c842f1ef87bb14eb04d06b4e14"}
Nov 02 09:47:30.434059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.434014    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f" event={"ID":"41246da8-0317-4f59-a90e-6b4f02b28115","Type":"ContainerStarted","Data":"e9bc9e71c3bcaebd97f6a962d9218063a0fe49132620a3db4a16657d10aa69f5"}
Nov 02 09:47:30.460287 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.460243    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:47:30.461539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.461517    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:47:30.462769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.462729    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f" podStartSLOduration=1.462717859 podStartE2EDuration="1.462717859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:30.461048882 +0000 UTC m=+727.399335657" watchObservedRunningTime="2026-11-02 09:47:30.462717859 +0000 UTC m=+727.401004635"
Nov 02 09:47:30.490402 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.490346    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-home\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.490856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.490489    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-tmp-dir\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.490856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.490586    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-model-cache\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.490856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.490625    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-dshm\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.490856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.490649    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z26mz\" (UniqueName: \"kubernetes.io/projected/e5827546-705c-4b91-a21e-daacfee4134a-kube-api-access-z26mz\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.490856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.490772    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5827546-705c-4b91-a21e-daacfee4134a-tls-certs\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.592000 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.591937    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-model-cache\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.592250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.592083    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-dshm\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.592250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.592138    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z26mz\" (UniqueName: \"kubernetes.io/projected/e5827546-705c-4b91-a21e-daacfee4134a-kube-api-access-z26mz\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.592250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.592245    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5827546-705c-4b91-a21e-daacfee4134a-tls-certs\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.592425 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.592301    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-home\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.592546 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.592518    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-tmp-dir\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.592787 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.592662    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-home\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.593249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.593196    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-tmp-dir\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.593452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.593394    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-model-cache\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.595011 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.594989    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-dshm\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.595142 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.595122    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5827546-705c-4b91-a21e-daacfee4134a-tls-certs\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.606485 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.606442    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z26mz\" (UniqueName: \"kubernetes.io/projected/e5827546-705c-4b91-a21e-daacfee4134a-kube-api-access-z26mz\") pod \"test-model-8743a5-kserve-59f899b667-rznvp\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") " pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.698763 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.698674    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:30.997984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:30.997949    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"]
Nov 02 09:47:31.010190 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:31.010161    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5827546_705c_4b91_a21e_daacfee4134a.slice/crio-9c84083ade0178ff9fc3b4c6752332ba880490ddae74214ddd0facf6857f59b6 WatchSource:0}: Error finding container 9c84083ade0178ff9fc3b4c6752332ba880490ddae74214ddd0facf6857f59b6: Status 404 returned error can't find the container with id 9c84083ade0178ff9fc3b4c6752332ba880490ddae74214ddd0facf6857f59b6
Nov 02 09:47:31.442506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:31.442463    4449 generic.go:356] "Generic (PLEG): container finished" podID="41246da8-0317-4f59-a90e-6b4f02b28115" containerID="3c6d78bda774cdc613412a3f52f9d3d7973758c842f1ef87bb14eb04d06b4e14" exitCode=0
Nov 02 09:47:31.442698 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:31.442555    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f" event={"ID":"41246da8-0317-4f59-a90e-6b4f02b28115","Type":"ContainerDied","Data":"3c6d78bda774cdc613412a3f52f9d3d7973758c842f1ef87bb14eb04d06b4e14"}
Nov 02 09:47:31.444366 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:31.444341    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" event={"ID":"e5827546-705c-4b91-a21e-daacfee4134a","Type":"ContainerStarted","Data":"ade4a05ea793c98d98c9ac01e9d1e26da578a965d19c1d0af5e2655a00177126"}
Nov 02 09:47:31.444499 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:31.444378    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" event={"ID":"e5827546-705c-4b91-a21e-daacfee4134a","Type":"ContainerStarted","Data":"9c84083ade0178ff9fc3b4c6752332ba880490ddae74214ddd0facf6857f59b6"}
Nov 02 09:47:31.483181 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:31.483128    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" podStartSLOduration=1.483113793 podStartE2EDuration="1.483113793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:31.479272886 +0000 UTC m=+728.417559688" watchObservedRunningTime="2026-11-02 09:47:31.483113793 +0000 UTC m=+728.421400621"
Nov 02 09:47:31.699386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:31.699283    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:32.615134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:32.615102    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"
Nov 02 09:47:32.721168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:32.721025    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41246da8-0317-4f59-a90e-6b4f02b28115-kube-api-access-wx7fk\" (UniqueName: \"kubernetes.io/projected/41246da8-0317-4f59-a90e-6b4f02b28115-kube-api-access-wx7fk\") pod \"41246da8-0317-4f59-a90e-6b4f02b28115\" (UID: \"41246da8-0317-4f59-a90e-6b4f02b28115\") "
Nov 02 09:47:32.721168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:32.721136    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-xks-ca\") pod \"41246da8-0317-4f59-a90e-6b4f02b28115\" (UID: \"41246da8-0317-4f59-a90e-6b4f02b28115\") "
Nov 02 09:47:32.722020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:32.721321    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-openshift-ca\") pod \"41246da8-0317-4f59-a90e-6b4f02b28115\" (UID: \"41246da8-0317-4f59-a90e-6b4f02b28115\") "
Nov 02 09:47:32.722020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:32.721736    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-openshift-ca" pod "41246da8-0317-4f59-a90e-6b4f02b28115" (UID: "41246da8-0317-4f59-a90e-6b4f02b28115"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:47:32.722020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:32.721862    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:32.724067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:32.724040    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41246da8-0317-4f59-a90e-6b4f02b28115-kube-api-access-wx7fk" pod "41246da8-0317-4f59-a90e-6b4f02b28115" (UID: "41246da8-0317-4f59-a90e-6b4f02b28115"). InnerVolumeSpecName "kube-api-access-wx7fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:32.724438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:32.724416    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-xks-ca" pod "41246da8-0317-4f59-a90e-6b4f02b28115" (UID: "41246da8-0317-4f59-a90e-6b4f02b28115"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:47:32.822977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:32.822945    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx7fk\" (UniqueName: \"kubernetes.io/projected/41246da8-0317-4f59-a90e-6b4f02b28115-kube-api-access-wx7fk\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:32.822977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:32.822975    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41246da8-0317-4f59-a90e-6b4f02b28115-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:33.460202 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:33.455626    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:33.513141 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:33.513053    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"
Nov 02 09:47:33.518243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:33.517026    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f" event={"ID":"41246da8-0317-4f59-a90e-6b4f02b28115","Type":"ContainerDied","Data":"e9bc9e71c3bcaebd97f6a962d9218063a0fe49132620a3db4a16657d10aa69f5"}
Nov 02 09:47:33.518243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:33.517083    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9bc9e71c3bcaebd97f6a962d9218063a0fe49132620a3db4a16657d10aa69f5"
Nov 02 09:47:33.574241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:33.567620    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:33.582181 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:33.582071    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:35.788442 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:35.788309    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7366-e69be8"]
Nov 02 09:47:35.789073 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:35.789053    4449 cpu_manager.go:414] "containerMap: removing container" podUID="41246da8-0317-4f59-a90e-6b4f02b28115" containerName="revoke-keys"
Nov 02 09:47:35.789279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:35.789076    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41246da8-0317-4f59-a90e-6b4f02b28115" containerName="revoke-keys"
Nov 02 09:47:35.789279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:35.789210    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="41246da8-0317-4f59-a90e-6b4f02b28115" containerName="revoke-keys"
Nov 02 09:47:35.793914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:35.793884    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-e69be8"
Nov 02 09:47:35.798313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:35.798283    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-e69be8"]
Nov 02 09:47:35.799180 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:35.798650    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8bqk\""
Nov 02 09:47:35.830896 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:35.829161    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tc7k\" (UniqueName: \"kubernetes.io/projected/9a77c7a9-16c1-436b-9ad9-cd401c273f8b-kube-api-access-6tc7k\") pod \"test-curl-7366-e69be8\" (UID: \"9a77c7a9-16c1-436b-9ad9-cd401c273f8b\") " pod="opendatahub/test-curl-7366-e69be8"
Nov 02 09:47:35.931082 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:35.931045    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tc7k\" (UniqueName: \"kubernetes.io/projected/9a77c7a9-16c1-436b-9ad9-cd401c273f8b-kube-api-access-6tc7k\") pod \"test-curl-7366-e69be8\" (UID: \"9a77c7a9-16c1-436b-9ad9-cd401c273f8b\") " pod="opendatahub/test-curl-7366-e69be8"
Nov 02 09:47:35.943253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:35.943188    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tc7k\" (UniqueName: \"kubernetes.io/projected/9a77c7a9-16c1-436b-9ad9-cd401c273f8b-kube-api-access-6tc7k\") pod \"test-curl-7366-e69be8\" (UID: \"9a77c7a9-16c1-436b-9ad9-cd401c273f8b\") " pod="opendatahub/test-curl-7366-e69be8"
Nov 02 09:47:36.121331 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:36.121248    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-e69be8"
Nov 02 09:47:36.397976 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:36.397945    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-e69be8"]
Nov 02 09:47:36.398271 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:36.398247    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a77c7a9_16c1_436b_9ad9_cd401c273f8b.slice/crio-ec90db81597d76ce573d5728e4697c384811c04c6b4ce74ec5a8fe631ee89c13 WatchSource:0}: Error finding container ec90db81597d76ce573d5728e4697c384811c04c6b4ce74ec5a8fe631ee89c13: Status 404 returned error can't find the container with id ec90db81597d76ce573d5728e4697c384811c04c6b4ce74ec5a8fe631ee89c13
Nov 02 09:47:36.552059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:36.552020    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7366-e69be8" event={"ID":"9a77c7a9-16c1-436b-9ad9-cd401c273f8b","Type":"ContainerStarted","Data":"e2ca0d0ca884b0d4778f1ba11af08ba0c20bf14609497e9101169adadabcc19b"}
Nov 02 09:47:36.552301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:36.552072    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7366-e69be8" event={"ID":"9a77c7a9-16c1-436b-9ad9-cd401c273f8b","Type":"ContainerStarted","Data":"ec90db81597d76ce573d5728e4697c384811c04c6b4ce74ec5a8fe631ee89c13"}
Nov 02 09:47:36.553585 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:36.553550    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:47:36.571489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:36.571440    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7366-e69be8" podStartSLOduration=1.571426426 podStartE2EDuration="1.571426426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:36.567638976 +0000 UTC m=+733.505925770" watchObservedRunningTime="2026-11-02 09:47:36.571426426 +0000 UTC m=+733.509713202"
Nov 02 09:47:37.289640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.289607    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7366-e69be8"]
Nov 02 09:47:37.294007 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.293976    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7366-e69be8"]
Nov 02 09:47:37.452661 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.452628    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7366-66627a"]
Nov 02 09:47:37.453376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.453356    4449 cpu_manager.go:414] "containerMap: removing container" podUID="9a77c7a9-16c1-436b-9ad9-cd401c273f8b" containerName="test-curl-7366-e69be8"
Nov 02 09:47:37.453507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.453378    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a77c7a9-16c1-436b-9ad9-cd401c273f8b" containerName="test-curl-7366-e69be8"
Nov 02 09:47:37.453566 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.453524    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a77c7a9-16c1-436b-9ad9-cd401c273f8b" containerName="test-curl-7366-e69be8"
Nov 02 09:47:37.456124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.456101    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:37.464439 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.464404    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-66627a"]
Nov 02 09:47:37.553744 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.553664    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmkd\" (UniqueName: \"kubernetes.io/projected/3fda788f-8862-410f-a7ed-6e20ba7d0a67-kube-api-access-xvmkd\") pod \"test-curl-7366-66627a\" (UID: \"3fda788f-8862-410f-a7ed-6e20ba7d0a67\") " pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:37.658210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.658165    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmkd\" (UniqueName: \"kubernetes.io/projected/3fda788f-8862-410f-a7ed-6e20ba7d0a67-kube-api-access-xvmkd\") pod \"test-curl-7366-66627a\" (UID: \"3fda788f-8862-410f-a7ed-6e20ba7d0a67\") " pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:37.690867 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.690835    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmkd\" (UniqueName: \"kubernetes.io/projected/3fda788f-8862-410f-a7ed-6e20ba7d0a67-kube-api-access-xvmkd\") pod \"test-curl-7366-66627a\" (UID: \"3fda788f-8862-410f-a7ed-6e20ba7d0a67\") " pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:37.790255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.788875    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:37.816407 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.814260    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"]
Nov 02 09:47:37.939694 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.939541    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"]
Nov 02 09:47:37.940251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.940191    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:37.945800 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.945562    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:37.951556 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.951521    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"]
Nov 02 09:47:37.956105 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:37.956077    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"]
Nov 02 09:47:38.040620 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.040584    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.040778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.040632    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20132bdb-f0fa-4408-9a26-5b34c1510038-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.040778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.040719    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4w6w\" (UniqueName: \"kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-kube-api-access-z4w6w\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.040778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.040769    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzqdw\" (UniqueName: \"kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-kube-api-access-tzqdw\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.040915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.040796    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.040915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.040823    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.040977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.040925    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.041027 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.041004    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.041114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.041098    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.041170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.041124    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f9a311-4b08-4384-8fbe-2b641f15880b-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.041267 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.041253    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.041337 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.041280    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.041388 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.041345    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.041448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.041423    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.041502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.041475    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.041558 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.041507    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.041611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.041557    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.041611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.041580    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.142570 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.142466    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f9a311-4b08-4384-8fbe-2b641f15880b-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.142572    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.142605    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.142628    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.142699    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.142750    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.142782    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.142836    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.142858    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.142899    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.142928    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20132bdb-f0fa-4408-9a26-5b34c1510038-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.142953    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4w6w\" (UniqueName: \"kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-kube-api-access-z4w6w\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.142982    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzqdw\" (UniqueName: \"kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-kube-api-access-tzqdw\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.143039    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.143086    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.143207    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.143266    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.143355    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.143453    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.143525    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.143776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.143549    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.144785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.143803    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.144785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.144091    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.144785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.144180    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f9a311-4b08-4384-8fbe-2b641f15880b-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.144785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.144091    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.144785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.144331    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20132bdb-f0fa-4408-9a26-5b34c1510038-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.144785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.144737    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.145889 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.145861    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.147117 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.147072    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.147253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.147200    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.147429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.147407    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.148522 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.148499    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.148675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.148654    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"]
Nov 02 09:47:38.152320 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.152305    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"
Nov 02 09:47:38.161170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.161011    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"]
Nov 02 09:47:38.161572 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.161539    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.164494 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.164176    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4w6w\" (UniqueName: \"kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-kube-api-access-z4w6w\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.166917 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.166893    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.167459 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.167401    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:47:38.167557 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.167489    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:47:38.169789 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.169763    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzqdw\" (UniqueName: \"kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-kube-api-access-tzqdw\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.218154 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.218119    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-66627a"]
Nov 02 09:47:38.218485 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:38.218453    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fda788f_8862_410f_a7ed_6e20ba7d0a67.slice/crio-6cc010e04a56ae11621ac2cf005a13e4b8818f3012965b7622d74e4a762240b5 WatchSource:0}: Error finding container 6cc010e04a56ae11621ac2cf005a13e4b8818f3012965b7622d74e4a762240b5: Status 404 returned error can't find the container with id 6cc010e04a56ae11621ac2cf005a13e4b8818f3012965b7622d74e4a762240b5
Nov 02 09:47:38.244149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.244121    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjp8x\" (UniqueName: \"kubernetes.io/projected/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-kube-api-access-gjp8x\") pod \"maas-api-revoke-sub-aa69fa05-snnv7\" (UID: \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"
Nov 02 09:47:38.244286 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.244171    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa69fa05-snnv7\" (UID: \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"
Nov 02 09:47:38.244512 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.244492    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa69fa05-snnv7\" (UID: \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"
Nov 02 09:47:38.248754 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.248733    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:47:38.248754 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.248749    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:47:38.283276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.283202    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:38.291672 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.290419    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:38.351551 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.346694    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa69fa05-snnv7\" (UID: \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"
Nov 02 09:47:38.351551 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.348605    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjp8x\" (UniqueName: \"kubernetes.io/projected/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-kube-api-access-gjp8x\") pod \"maas-api-revoke-sub-aa69fa05-snnv7\" (UID: \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"
Nov 02 09:47:38.351551 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.348659    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa69fa05-snnv7\" (UID: \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"
Nov 02 09:47:38.351551 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.348347    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa69fa05-snnv7\" (UID: \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"
Nov 02 09:47:38.353869 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.351831    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa69fa05-snnv7\" (UID: \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"
Nov 02 09:47:38.371262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.370563    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjp8x\" (UniqueName: \"kubernetes.io/projected/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-kube-api-access-gjp8x\") pod \"maas-api-revoke-sub-aa69fa05-snnv7\" (UID: \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"
Nov 02 09:47:38.489582 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.489508    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"
Nov 02 09:47:38.579312 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.579274    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"]
Nov 02 09:47:38.580494 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:38.580140    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20132bdb_f0fa_4408_9a26_5b34c1510038.slice/crio-08e948fd33380a7b5b3cd5b3d37ddd1a39e73894a72710441a9c2c7ce648155c WatchSource:0}: Error finding container 08e948fd33380a7b5b3cd5b3d37ddd1a39e73894a72710441a9c2c7ce648155c: Status 404 returned error can't find the container with id 08e948fd33380a7b5b3cd5b3d37ddd1a39e73894a72710441a9c2c7ce648155c
Nov 02 09:47:38.593243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.592685    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:47:38.593243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.592901    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:47:38.593243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.592996    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:47:38.598386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.598351    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"]
Nov 02 09:47:38.603870 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:38.603196    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f9a311_4b08_4384_8fbe_2b641f15880b.slice/crio-bf5e0e1017b81b15ac126d5c46754c36abf3491d2817ac2d9eaf5139aff42bf9 WatchSource:0}: Error finding container bf5e0e1017b81b15ac126d5c46754c36abf3491d2817ac2d9eaf5139aff42bf9: Status 404 returned error can't find the container with id bf5e0e1017b81b15ac126d5c46754c36abf3491d2817ac2d9eaf5139aff42bf9
Nov 02 09:47:38.604267 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.604146    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7366-66627a" event={"ID":"3fda788f-8862-410f-a7ed-6e20ba7d0a67","Type":"ContainerStarted","Data":"b2a877da26117a3d6c841111b6f55bcaa67dbaf13fbbbdb47a0b752627ee09d4"}
Nov 02 09:47:38.604267 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.604188    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7366-66627a" event={"ID":"3fda788f-8862-410f-a7ed-6e20ba7d0a67","Type":"ContainerStarted","Data":"6cc010e04a56ae11621ac2cf005a13e4b8818f3012965b7622d74e4a762240b5"}
Nov 02 09:47:38.607265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.604394    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7366-e69be8" podUID="9a77c7a9-16c1-436b-9ad9-cd401c273f8b" containerName="test-curl-7366-e69be8" containerID="cri-o://e2ca0d0ca884b0d4778f1ba11af08ba0c20bf14609497e9101169adadabcc19b" gracePeriod=2
Nov 02 09:47:38.607850 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:38.607819    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-e69be8\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="9a77c7a9-16c1-436b-9ad9-cd401c273f8b" pod="opendatahub/test-curl-7366-e69be8"
Nov 02 09:47:38.612772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.612730    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:47:38.612928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.612819    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:47:38.612928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.612862    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:47:38.632645 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:38.632358    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-e69be8\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="9a77c7a9-16c1-436b-9ad9-cd401c273f8b" pod="opendatahub/test-curl-7366-e69be8"
Nov 02 09:47:38.823764 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.820879    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7366-66627a" podStartSLOduration=1.820860506 podStartE2EDuration="1.820860506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:38.629532946 +0000 UTC m=+735.567819740" watchObservedRunningTime="2026-11-02 09:47:38.820860506 +0000 UTC m=+735.759147284"
Nov 02 09:47:38.842275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:38.836088    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"]
Nov 02 09:47:39.618308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.618171    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7366-66627a"]
Nov 02 09:47:39.623244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.623047    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7366-66627a"]
Nov 02 09:47:39.632534 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.632501    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" event={"ID":"20132bdb-f0fa-4408-9a26-5b34c1510038","Type":"ContainerStarted","Data":"e6f02ebcdcac78aceffa25695b576b3034ffcb0aa94746384ed7bd0bcd6a3ac5"}
Nov 02 09:47:39.632704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.632562    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" event={"ID":"20132bdb-f0fa-4408-9a26-5b34c1510038","Type":"ContainerStarted","Data":"08e948fd33380a7b5b3cd5b3d37ddd1a39e73894a72710441a9c2c7ce648155c"}
Nov 02 09:47:39.635638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.635555    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"]
Nov 02 09:47:39.636252 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.636209    4449 cpu_manager.go:414] "containerMap: removing container" podUID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" containerName="test-curl-7366-66627a"
Nov 02 09:47:39.636252 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.636248    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" containerName="test-curl-7366-66627a"
Nov 02 09:47:39.636419 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.636394    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" containerName="test-curl-7366-66627a"
Nov 02 09:47:39.640072 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.640036    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7" event={"ID":"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0","Type":"ContainerStarted","Data":"64822bafa9166e63c90286251132b14ce08c46133a581f42f8f3f236f1f1f8af"}
Nov 02 09:47:39.640189 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.640087    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7" event={"ID":"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0","Type":"ContainerStarted","Data":"4bf0bf978da620bad06fea5ac9a6f5348946b9be6c4fe4bd1166e79f34857921"}
Nov 02 09:47:39.640189 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.640181    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"
Nov 02 09:47:39.642070 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.642002    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" event={"ID":"09f9a311-4b08-4384-8fbe-2b641f15880b","Type":"ContainerStarted","Data":"eee7d9cba4cfe5574e4893f6da68f0097b6872d95745e4271778dc19318463cc"}
Nov 02 09:47:39.642070 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.642053    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" event={"ID":"09f9a311-4b08-4384-8fbe-2b641f15880b","Type":"ContainerStarted","Data":"bf5e0e1017b81b15ac126d5c46754c36abf3491d2817ac2d9eaf5139aff42bf9"}
Nov 02 09:47:39.648041 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.648014    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"]
Nov 02 09:47:39.686241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.675235    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" podStartSLOduration=2.67519293 podStartE2EDuration="2.67519293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:39.673953218 +0000 UTC m=+736.612240057" watchObservedRunningTime="2026-11-02 09:47:39.67519293 +0000 UTC m=+736.613479706"
Nov 02 09:47:39.718843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.718787    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7" podStartSLOduration=1.71877345 podStartE2EDuration="1.71877345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:39.698844496 +0000 UTC m=+736.637131274" watchObservedRunningTime="2026-11-02 09:47:39.71877345 +0000 UTC m=+736.657060225"
Nov 02 09:47:39.729025 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.728782    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7366-f11c97"]
Nov 02 09:47:39.733038 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.733012    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-f11c97"
Nov 02 09:47:39.743859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.743830    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-f11c97"]
Nov 02 09:47:39.747466 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.747412    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" podStartSLOduration=2.747395235 podStartE2EDuration="2.747395235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:39.740254225 +0000 UTC m=+736.678541025" watchObservedRunningTime="2026-11-02 09:47:39.747395235 +0000 UTC m=+736.685682012"
Nov 02 09:47:39.817193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.817135    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb14f2e1-vnqkk\" (UID: \"226bd2c7-0279-4dad-8b4a-512d5f30588e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"
Nov 02 09:47:39.817427 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.817202    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb14f2e1-vnqkk\" (UID: \"226bd2c7-0279-4dad-8b4a-512d5f30588e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"
Nov 02 09:47:39.817427 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.817249    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mxw5\" (UniqueName: \"kubernetes.io/projected/226bd2c7-0279-4dad-8b4a-512d5f30588e-kube-api-access-5mxw5\") pod \"maas-api-revoke-sub-cb14f2e1-vnqkk\" (UID: \"226bd2c7-0279-4dad-8b4a-512d5f30588e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"
Nov 02 09:47:39.918919 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.918828    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb14f2e1-vnqkk\" (UID: \"226bd2c7-0279-4dad-8b4a-512d5f30588e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"
Nov 02 09:47:39.918919 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.918873    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mxw5\" (UniqueName: \"kubernetes.io/projected/226bd2c7-0279-4dad-8b4a-512d5f30588e-kube-api-access-5mxw5\") pod \"maas-api-revoke-sub-cb14f2e1-vnqkk\" (UID: \"226bd2c7-0279-4dad-8b4a-512d5f30588e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"
Nov 02 09:47:39.919160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.919030    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxkcw\" (UniqueName: \"kubernetes.io/projected/10350da6-ad59-4afe-a58a-91ff5ae83f4a-kube-api-access-pxkcw\") pod \"test-curl-7366-f11c97\" (UID: \"10350da6-ad59-4afe-a58a-91ff5ae83f4a\") " pod="opendatahub/test-curl-7366-f11c97"
Nov 02 09:47:39.919330 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.919313    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb14f2e1-vnqkk\" (UID: \"226bd2c7-0279-4dad-8b4a-512d5f30588e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"
Nov 02 09:47:39.919933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.919913    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb14f2e1-vnqkk\" (UID: \"226bd2c7-0279-4dad-8b4a-512d5f30588e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"
Nov 02 09:47:39.921443 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.921425    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb14f2e1-vnqkk\" (UID: \"226bd2c7-0279-4dad-8b4a-512d5f30588e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"
Nov 02 09:47:39.931324 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:39.931305    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mxw5\" (UniqueName: \"kubernetes.io/projected/226bd2c7-0279-4dad-8b4a-512d5f30588e-kube-api-access-5mxw5\") pod \"maas-api-revoke-sub-cb14f2e1-vnqkk\" (UID: \"226bd2c7-0279-4dad-8b4a-512d5f30588e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"
Nov 02 09:47:40.018990 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.018962    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"
Nov 02 09:47:40.020754 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.020727    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxkcw\" (UniqueName: \"kubernetes.io/projected/10350da6-ad59-4afe-a58a-91ff5ae83f4a-kube-api-access-pxkcw\") pod \"test-curl-7366-f11c97\" (UID: \"10350da6-ad59-4afe-a58a-91ff5ae83f4a\") " pod="opendatahub/test-curl-7366-f11c97"
Nov 02 09:47:40.033719 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.033697    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxkcw\" (UniqueName: \"kubernetes.io/projected/10350da6-ad59-4afe-a58a-91ff5ae83f4a-kube-api-access-pxkcw\") pod \"test-curl-7366-f11c97\" (UID: \"10350da6-ad59-4afe-a58a-91ff5ae83f4a\") " pod="opendatahub/test-curl-7366-f11c97"
Nov 02 09:47:40.064557 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.063439    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-f11c97"
Nov 02 09:47:40.305432 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.301782    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 09:47:40.305432 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.301842    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" podUID="20132bdb-f0fa-4408-9a26-5b34c1510038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 09:47:40.305432 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.302126    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:40.305432 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.304828    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:40.305988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.305893    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 09:47:40.305988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.305936    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 09:47:40.405579 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.405104    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"]
Nov 02 09:47:40.408759 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:40.408722    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226bd2c7_0279_4dad_8b4a_512d5f30588e.slice/crio-da52ff24649755f3aa7140dfd1e7b6f073b9224c1ef9cda04042c789a2e300cc WatchSource:0}: Error finding container da52ff24649755f3aa7140dfd1e7b6f073b9224c1ef9cda04042c789a2e300cc: Status 404 returned error can't find the container with id da52ff24649755f3aa7140dfd1e7b6f073b9224c1ef9cda04042c789a2e300cc
Nov 02 09:47:40.448768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.448712    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-f11c97"]
Nov 02 09:47:40.700290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.697908    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7366-f11c97" event={"ID":"10350da6-ad59-4afe-a58a-91ff5ae83f4a","Type":"ContainerStarted","Data":"130cc99d15e38e7a421120c19803e23c8ea6a054633af47d897351f042f94969"}
Nov 02 09:47:40.700290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.697961    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7366-f11c97" event={"ID":"10350da6-ad59-4afe-a58a-91ff5ae83f4a","Type":"ContainerStarted","Data":"93b46d0b6e5febcde2ba81b423c93217b23d45d2f16fe79b61b3064942445e7c"}
Nov 02 09:47:40.704657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.703305    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:40.708197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.708169    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk" event={"ID":"226bd2c7-0279-4dad-8b4a-512d5f30588e","Type":"ContainerStarted","Data":"da52ff24649755f3aa7140dfd1e7b6f073b9224c1ef9cda04042c789a2e300cc"}
Nov 02 09:47:40.710120 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.710095    4449 generic.go:356] "Generic (PLEG): container finished" podID="9a77c7a9-16c1-436b-9ad9-cd401c273f8b" containerID="e2ca0d0ca884b0d4778f1ba11af08ba0c20bf14609497e9101169adadabcc19b" exitCode=137
Nov 02 09:47:40.712352 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.712324    4449 generic.go:356] "Generic (PLEG): container finished" podID="ef689a32-1555-4d9f-93d9-9c94fdb1d8f0" containerID="64822bafa9166e63c90286251132b14ce08c46133a581f42f8f3f236f1f1f8af" exitCode=0
Nov 02 09:47:40.712475 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.712399    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7" event={"ID":"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0","Type":"ContainerDied","Data":"64822bafa9166e63c90286251132b14ce08c46133a581f42f8f3f236f1f1f8af"}
Nov 02 09:47:40.713068 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.712758    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7366-66627a" podUID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" containerName="test-curl-7366-66627a" containerID="cri-o://b2a877da26117a3d6c841111b6f55bcaa67dbaf13fbbbdb47a0b752627ee09d4" gracePeriod=2
Nov 02 09:47:40.719562 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.719264    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:40.721562 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:40.721534    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-66627a\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:40.722778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:40.722742    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7366-f11c97" podStartSLOduration=1.72273137 podStartE2EDuration="1.72273137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:40.719001786 +0000 UTC m=+737.657288558" watchObservedRunningTime="2026-11-02 09:47:40.72273137 +0000 UTC m=+737.661018146"
Nov 02 09:47:40.789182 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:40.789126    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-66627a\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:41.002650 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.002273    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-e69be8"
Nov 02 09:47:41.005954 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:41.005557    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-66627a\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:41.038647 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.038624    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a77c7a9-16c1-436b-9ad9-cd401c273f8b-kube-api-access-6tc7k\" (UniqueName: \"kubernetes.io/projected/9a77c7a9-16c1-436b-9ad9-cd401c273f8b-kube-api-access-6tc7k\") pod \"9a77c7a9-16c1-436b-9ad9-cd401c273f8b\" (UID: \"9a77c7a9-16c1-436b-9ad9-cd401c273f8b\") "
Nov 02 09:47:41.040931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.040911    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a77c7a9-16c1-436b-9ad9-cd401c273f8b-kube-api-access-6tc7k" pod "9a77c7a9-16c1-436b-9ad9-cd401c273f8b" (UID: "9a77c7a9-16c1-436b-9ad9-cd401c273f8b"). InnerVolumeSpecName "kube-api-access-6tc7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:41.140622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.140587    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tc7k\" (UniqueName: \"kubernetes.io/projected/9a77c7a9-16c1-436b-9ad9-cd401c273f8b-kube-api-access-6tc7k\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:41.208882 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.208852    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a77c7a9-16c1-436b-9ad9-cd401c273f8b" path="/var/lib/kubelet/pods/9a77c7a9-16c1-436b-9ad9-cd401c273f8b/volumes"
Nov 02 09:47:41.298775 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.298320    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 09:47:41.298775 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.298379    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 09:47:41.301207 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.301019    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 09:47:41.301207 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.301086    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" podUID="20132bdb-f0fa-4408-9a26-5b34c1510038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 09:47:41.513538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.513456    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7366-f11c97"]
Nov 02 09:47:41.519375 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.519345    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7366-f11c97"]
Nov 02 09:47:41.706947 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.706912    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7366-0c0970"]
Nov 02 09:47:41.707958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.707932    4449 cpu_manager.go:414] "containerMap: removing container" podUID="10350da6-ad59-4afe-a58a-91ff5ae83f4a" containerName="test-curl-7366-f11c97"
Nov 02 09:47:41.707958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.707958    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10350da6-ad59-4afe-a58a-91ff5ae83f4a" containerName="test-curl-7366-f11c97"
Nov 02 09:47:41.708151 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.708140    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="10350da6-ad59-4afe-a58a-91ff5ae83f4a" containerName="test-curl-7366-f11c97"
Nov 02 09:47:41.712146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.712082    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-0c0970"
Nov 02 09:47:41.713636 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.713549    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-0c0970"]
Nov 02 09:47:41.721250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.720960    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:41.729360 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.726562    4449 generic.go:356] "Generic (PLEG): container finished" podID="226bd2c7-0279-4dad-8b4a-512d5f30588e" containerID="dfe631ec74ef3a6f9100b9eb17b1ef7678141315e47f65d4909a7752377ce239" exitCode=0
Nov 02 09:47:41.729360 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.726712    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk" event={"ID":"226bd2c7-0279-4dad-8b4a-512d5f30588e","Type":"ContainerDied","Data":"dfe631ec74ef3a6f9100b9eb17b1ef7678141315e47f65d4909a7752377ce239"}
Nov 02 09:47:41.731928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.731907    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-e69be8"
Nov 02 09:47:41.732053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.732040    4449 scope.go:122] "RemoveContainer" containerID="e2ca0d0ca884b0d4778f1ba11af08ba0c20bf14609497e9101169adadabcc19b"
Nov 02 09:47:41.775248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.773935    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmnjr\" (UniqueName: \"kubernetes.io/projected/767250c5-1f49-4bc9-b77f-0d2c8d55685e-kube-api-access-tmnjr\") pod \"test-curl-7366-0c0970\" (UID: \"767250c5-1f49-4bc9-b77f-0d2c8d55685e\") " pod="opendatahub/test-curl-7366-0c0970"
Nov 02 09:47:41.920240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.919521    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmnjr\" (UniqueName: \"kubernetes.io/projected/767250c5-1f49-4bc9-b77f-0d2c8d55685e-kube-api-access-tmnjr\") pod \"test-curl-7366-0c0970\" (UID: \"767250c5-1f49-4bc9-b77f-0d2c8d55685e\") " pod="opendatahub/test-curl-7366-0c0970"
Nov 02 09:47:41.944250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:41.943796    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmnjr\" (UniqueName: \"kubernetes.io/projected/767250c5-1f49-4bc9-b77f-0d2c8d55685e-kube-api-access-tmnjr\") pod \"test-curl-7366-0c0970\" (UID: \"767250c5-1f49-4bc9-b77f-0d2c8d55685e\") " pod="opendatahub/test-curl-7366-0c0970"
Nov 02 09:47:42.033354 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.032428    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-0c0970"
Nov 02 09:47:42.146037 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.146003    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"
Nov 02 09:47:42.292944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.292901    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 09:47:42.293109 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.292972    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 09:47:42.293451 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.293429    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 09:47:42.293571 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.293469    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" podUID="20132bdb-f0fa-4408-9a26-5b34c1510038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 09:47:42.328746 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.328709    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-openshift-ca\") pod \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\" (UID: \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\") "
Nov 02 09:47:42.328936 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.328876    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-xks-ca\") pod \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\" (UID: \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\") "
Nov 02 09:47:42.328997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.328955    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-kube-api-access-gjp8x\" (UniqueName: \"kubernetes.io/projected/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-kube-api-access-gjp8x\") pod \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\" (UID: \"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0\") "
Nov 02 09:47:42.330133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.329593    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-openshift-ca" pod "ef689a32-1555-4d9f-93d9-9c94fdb1d8f0" (UID: "ef689a32-1555-4d9f-93d9-9c94fdb1d8f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:47:42.330133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.330074    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:42.332366 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.332320    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-xks-ca" pod "ef689a32-1555-4d9f-93d9-9c94fdb1d8f0" (UID: "ef689a32-1555-4d9f-93d9-9c94fdb1d8f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:47:42.332500 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.332356    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-kube-api-access-gjp8x" pod "ef689a32-1555-4d9f-93d9-9c94fdb1d8f0" (UID: "ef689a32-1555-4d9f-93d9-9c94fdb1d8f0"). InnerVolumeSpecName "kube-api-access-gjp8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:42.426337 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:42.426305    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod767250c5_1f49_4bc9_b77f_0d2c8d55685e.slice/crio-e05c90286e2e28b10de3ee1218688e7965e6cadc50c637126702be8901f3f8c0 WatchSource:0}: Error finding container e05c90286e2e28b10de3ee1218688e7965e6cadc50c637126702be8901f3f8c0: Status 404 returned error can't find the container with id e05c90286e2e28b10de3ee1218688e7965e6cadc50c637126702be8901f3f8c0
Nov 02 09:47:42.426435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.426412    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-0c0970"]
Nov 02 09:47:42.430707 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.430689    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:42.430759 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.430711    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjp8x\" (UniqueName: \"kubernetes.io/projected/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0-kube-api-access-gjp8x\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:42.741273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.741244    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"
Nov 02 09:47:42.741706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.741243    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7" event={"ID":"ef689a32-1555-4d9f-93d9-9c94fdb1d8f0","Type":"ContainerDied","Data":"4bf0bf978da620bad06fea5ac9a6f5348946b9be6c4fe4bd1166e79f34857921"}
Nov 02 09:47:42.741706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.741381    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bf0bf978da620bad06fea5ac9a6f5348946b9be6c4fe4bd1166e79f34857921"
Nov 02 09:47:42.743393 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.743369    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7366-0c0970" event={"ID":"767250c5-1f49-4bc9-b77f-0d2c8d55685e","Type":"ContainerStarted","Data":"4acf272b87476c30e4b47278bdac8fc1bc3ba7a7c434bdd9d839c96444779c23"}
Nov 02 09:47:42.743517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.743408    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7366-0c0970" event={"ID":"767250c5-1f49-4bc9-b77f-0d2c8d55685e","Type":"ContainerStarted","Data":"e05c90286e2e28b10de3ee1218688e7965e6cadc50c637126702be8901f3f8c0"}
Nov 02 09:47:42.745076 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.745053    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7366-f11c97" podUID="10350da6-ad59-4afe-a58a-91ff5ae83f4a" containerName="test-curl-7366-f11c97" containerID="cri-o://130cc99d15e38e7a421120c19803e23c8ea6a054633af47d897351f042f94969" gracePeriod=2
Nov 02 09:47:42.799243 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:42.795554    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-f11c97\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="10350da6-ad59-4afe-a58a-91ff5ae83f4a" pod="opendatahub/test-curl-7366-f11c97"
Nov 02 09:47:42.806439 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:42.806395    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-f11c97\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="10350da6-ad59-4afe-a58a-91ff5ae83f4a" pod="opendatahub/test-curl-7366-f11c97"
Nov 02 09:47:42.806600 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:42.806502    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7366-0c0970" podStartSLOduration=1.80648219 podStartE2EDuration="1.80648219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:42.792880629 +0000 UTC m=+739.731167434" watchObservedRunningTime="2026-11-02 09:47:42.80648219 +0000 UTC m=+739.744768966"
Nov 02 09:47:43.124119 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.121419    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"
Nov 02 09:47:43.142131 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:43.142091    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-f11c97\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="10350da6-ad59-4afe-a58a-91ff5ae83f4a" pod="opendatahub/test-curl-7366-f11c97"
Nov 02 09:47:43.230841 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:43.230763    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-f11c97\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="10350da6-ad59-4afe-a58a-91ff5ae83f4a" pod="opendatahub/test-curl-7366-f11c97"
Nov 02 09:47:43.239233 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.239191    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-openshift-ca\") pod \"226bd2c7-0279-4dad-8b4a-512d5f30588e\" (UID: \"226bd2c7-0279-4dad-8b4a-512d5f30588e\") "
Nov 02 09:47:43.239333 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.239256    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-xks-ca\") pod \"226bd2c7-0279-4dad-8b4a-512d5f30588e\" (UID: \"226bd2c7-0279-4dad-8b4a-512d5f30588e\") "
Nov 02 09:47:43.239333 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.239317    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226bd2c7-0279-4dad-8b4a-512d5f30588e-kube-api-access-5mxw5\" (UniqueName: \"kubernetes.io/projected/226bd2c7-0279-4dad-8b4a-512d5f30588e-kube-api-access-5mxw5\") pod \"226bd2c7-0279-4dad-8b4a-512d5f30588e\" (UID: \"226bd2c7-0279-4dad-8b4a-512d5f30588e\") "
Nov 02 09:47:43.239616 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.239588    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-openshift-ca" pod "226bd2c7-0279-4dad-8b4a-512d5f30588e" (UID: "226bd2c7-0279-4dad-8b4a-512d5f30588e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:47:43.239819 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.239803    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:43.241858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.241836    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226bd2c7-0279-4dad-8b4a-512d5f30588e-kube-api-access-5mxw5" pod "226bd2c7-0279-4dad-8b4a-512d5f30588e" (UID: "226bd2c7-0279-4dad-8b4a-512d5f30588e"). InnerVolumeSpecName "kube-api-access-5mxw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:43.241858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.241849    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-xks-ca" pod "226bd2c7-0279-4dad-8b4a-512d5f30588e" (UID: "226bd2c7-0279-4dad-8b4a-512d5f30588e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:47:43.249089 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.249061    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:43.272251 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:43.257472    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-66627a\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:43.295629 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.295120    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 09:47:43.295629 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.295179    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" podUID="20132bdb-f0fa-4408-9a26-5b34c1510038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 09:47:43.295977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.295954    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 09:47:43.296111 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.296090    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 09:47:43.341100 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.340768    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fda788f-8862-410f-a7ed-6e20ba7d0a67-kube-api-access-xvmkd\" (UniqueName: \"kubernetes.io/projected/3fda788f-8862-410f-a7ed-6e20ba7d0a67-kube-api-access-xvmkd\") pod \"3fda788f-8862-410f-a7ed-6e20ba7d0a67\" (UID: \"3fda788f-8862-410f-a7ed-6e20ba7d0a67\") "
Nov 02 09:47:43.341100 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.341065    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mxw5\" (UniqueName: \"kubernetes.io/projected/226bd2c7-0279-4dad-8b4a-512d5f30588e-kube-api-access-5mxw5\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:43.341100 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.341076    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/226bd2c7-0279-4dad-8b4a-512d5f30588e-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:43.343071 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.343047    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fda788f-8862-410f-a7ed-6e20ba7d0a67-kube-api-access-xvmkd" pod "3fda788f-8862-410f-a7ed-6e20ba7d0a67" (UID: "3fda788f-8862-410f-a7ed-6e20ba7d0a67"). InnerVolumeSpecName "kube-api-access-xvmkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:43.442140 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.442109    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvmkd\" (UniqueName: \"kubernetes.io/projected/3fda788f-8862-410f-a7ed-6e20ba7d0a67-kube-api-access-xvmkd\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:43.710243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.709369    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7366-0c0970"]
Nov 02 09:47:43.716599 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.716545    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7366-0c0970"]
Nov 02 09:47:43.812844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.812810    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk" event={"ID":"226bd2c7-0279-4dad-8b4a-512d5f30588e","Type":"ContainerDied","Data":"da52ff24649755f3aa7140dfd1e7b6f073b9224c1ef9cda04042c789a2e300cc"}
Nov 02 09:47:43.813195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.812865    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da52ff24649755f3aa7140dfd1e7b6f073b9224c1ef9cda04042c789a2e300cc"
Nov 02 09:47:43.813195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.812928    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"
Nov 02 09:47:43.820511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.820476    4449 generic.go:356] "Generic (PLEG): container finished" podID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" containerID="b2a877da26117a3d6c841111b6f55bcaa67dbaf13fbbbdb47a0b752627ee09d4" exitCode=137
Nov 02 09:47:43.820686 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.820529    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:43.820686 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.820573    4449 scope.go:122] "RemoveContainer" containerID="b2a877da26117a3d6c841111b6f55bcaa67dbaf13fbbbdb47a0b752627ee09d4"
Nov 02 09:47:43.825018 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:43.824966    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-66627a\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:43.837209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.836686    4449 scope.go:122] "RemoveContainer" containerID="b2a877da26117a3d6c841111b6f55bcaa67dbaf13fbbbdb47a0b752627ee09d4"
Nov 02 09:47:43.837388 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:43.837359    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a877da26117a3d6c841111b6f55bcaa67dbaf13fbbbdb47a0b752627ee09d4\": container with ID starting with b2a877da26117a3d6c841111b6f55bcaa67dbaf13fbbbdb47a0b752627ee09d4 not found: ID does not exist" containerID="b2a877da26117a3d6c841111b6f55bcaa67dbaf13fbbbdb47a0b752627ee09d4"
Nov 02 09:47:43.837453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:43.837401    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a877da26117a3d6c841111b6f55bcaa67dbaf13fbbbdb47a0b752627ee09d4"} err="failed to get container status \"b2a877da26117a3d6c841111b6f55bcaa67dbaf13fbbbdb47a0b752627ee09d4\": rpc error: code = NotFound desc = could not find container \"b2a877da26117a3d6c841111b6f55bcaa67dbaf13fbbbdb47a0b752627ee09d4\": container with ID starting with b2a877da26117a3d6c841111b6f55bcaa67dbaf13fbbbdb47a0b752627ee09d4 not found: ID does not exist"
Nov 02 09:47:43.837519 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:43.837487    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-66627a\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:43.841307 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:43.841274    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-66627a\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:44.293242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:44.292551    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 09:47:44.293242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:44.292609    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" podUID="20132bdb-f0fa-4408-9a26-5b34c1510038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 09:47:44.293526 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:44.293305    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 09:47:44.293526 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:44.293347    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 09:47:44.905245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:44.891147    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7366-0c0970" podUID="767250c5-1f49-4bc9-b77f-0d2c8d55685e" containerName="test-curl-7366-0c0970" containerID="cri-o://4acf272b87476c30e4b47278bdac8fc1bc3ba7a7c434bdd9d839c96444779c23" gracePeriod=2
Nov 02 09:47:44.905245 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:44.894579    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-0c0970\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="767250c5-1f49-4bc9-b77f-0d2c8d55685e" pod="opendatahub/test-curl-7366-0c0970"
Nov 02 09:47:44.905245 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:44.900479    4449 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-66627a\" is forbidden: User \"system:node:ip-10-0-128-217.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-217.us-east-2.compute.internal' and this object" podUID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" pod="opendatahub/test-curl-7366-66627a"
Nov 02 09:47:45.251029 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.250556    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fda788f-8862-410f-a7ed-6e20ba7d0a67" path="/var/lib/kubelet/pods/3fda788f-8862-410f-a7ed-6e20ba7d0a67/volumes"
Nov 02 09:47:45.291970 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.291934    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 09:47:45.292277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.292232    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" podUID="20132bdb-f0fa-4408-9a26-5b34c1510038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 09:47:45.293263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.293241    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 09:47:45.293461 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.293432    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 09:47:45.428303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.426296    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"]
Nov 02 09:47:45.428303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.427310    4449 cpu_manager.go:414] "containerMap: removing container" podUID="767250c5-1f49-4bc9-b77f-0d2c8d55685e" containerName="test-curl-7366-0c0970"
Nov 02 09:47:45.428303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.427332    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767250c5-1f49-4bc9-b77f-0d2c8d55685e" containerName="test-curl-7366-0c0970"
Nov 02 09:47:45.428303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.427420    4449 cpu_manager.go:414] "containerMap: removing container" podUID="226bd2c7-0279-4dad-8b4a-512d5f30588e" containerName="revoke-keys"
Nov 02 09:47:45.428303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.427430    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226bd2c7-0279-4dad-8b4a-512d5f30588e" containerName="revoke-keys"
Nov 02 09:47:45.428303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.427444    4449 cpu_manager.go:414] "containerMap: removing container" podUID="ef689a32-1555-4d9f-93d9-9c94fdb1d8f0" containerName="revoke-keys"
Nov 02 09:47:45.428303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.427465    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef689a32-1555-4d9f-93d9-9c94fdb1d8f0" containerName="revoke-keys"
Nov 02 09:47:45.428303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.427630    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="226bd2c7-0279-4dad-8b4a-512d5f30588e" containerName="revoke-keys"
Nov 02 09:47:45.428303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.427644    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef689a32-1555-4d9f-93d9-9c94fdb1d8f0" containerName="revoke-keys"
Nov 02 09:47:45.428303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.427656    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="767250c5-1f49-4bc9-b77f-0d2c8d55685e" containerName="test-curl-7366-0c0970"
Nov 02 09:47:45.576310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.562312    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.577666 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.577627    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-dshm\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.653271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.639693    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-model-cache\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.653271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.639748    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tmp-dir\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.653271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.639917    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-home\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.653271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.639948    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tls-certs\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.653271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.640000    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qcnf\" (UniqueName: \"kubernetes.io/projected/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-kube-api-access-2qcnf\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.653271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.580359    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"]
Nov 02 09:47:45.653271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.581600    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz" podUID="5ed165dd-1500-4fd4-a5ae-5a35d894726e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:45.721714 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.719720    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-bde944\"/\"rate-test-model-bde944-kserve-self-signed-certs\""
Nov 02 09:47:45.721714 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.720811    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-bde944\"/\"openshift-service-ca.crt\""
Nov 02 09:47:45.721714 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.720906    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-bde944\"/\"kube-root-ca.crt\""
Nov 02 09:47:45.721714 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.721003    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-bde944\"/\"default-dockercfg-4ktdd\""
Nov 02 09:47:45.721714 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.721058    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" podUID="e5827546-705c-4b91-a21e-daacfee4134a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:45.826242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.812382    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-model-cache\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.826242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.812426    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tmp-dir\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.826242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.812568    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-home\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.826242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.812598    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tls-certs\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.826242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.812647    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qcnf\" (UniqueName: \"kubernetes.io/projected/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-kube-api-access-2qcnf\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.826242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.812733    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-dshm\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.857430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.843837    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-model-cache\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.857430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.844118    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tmp-dir\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.898506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.861456    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:45.898506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.861889    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-home\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:45.900347 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:45.899631    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-dshm\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:46.101525 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:46.101487    4449 generic.go:356] "Generic (PLEG): container finished" podID="10350da6-ad59-4afe-a58a-91ff5ae83f4a" containerID="130cc99d15e38e7a421120c19803e23c8ea6a054633af47d897351f042f94969" exitCode=137
Nov 02 09:47:46.142384 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:46.125444    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tls-certs\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:46.142384 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:46.129875    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qcnf\" (UniqueName: \"kubernetes.io/projected/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-kube-api-access-2qcnf\") pod \"rate-test-model-bde944-kserve-5df8884f97-6xflz\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") " pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:46.301245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:46.286480    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:46.329789 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:46.326704    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 09:47:46.329789 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:46.326761    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 09:47:46.329789 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:46.327199    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 09:47:46.329789 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:46.327263    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" podUID="20132bdb-f0fa-4408-9a26-5b34c1510038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 09:47:46.640416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:46.640366    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz" podUID="5ed165dd-1500-4fd4-a5ae-5a35d894726e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 02 09:47:46.642756 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:46.642727    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:46.727795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:46.727694    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" podUID="e5827546-705c-4b91-a21e-daacfee4134a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:46.729137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:46.729111    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:46.975399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:46.975349    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:47.025297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:47.023337    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 09:47:47.025297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:47.023465    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:47:47.737890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:47.737769    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz" podUID="5ed165dd-1500-4fd4-a5ae-5a35d894726e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:47.887678 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:47.887627    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" podUID="e5827546-705c-4b91-a21e-daacfee4134a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:48.056430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.056379    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:48.231876 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.231586    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:48.399259 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.398911    4449 generic.go:356] "Generic (PLEG): container finished" podID="767250c5-1f49-4bc9-b77f-0d2c8d55685e" containerID="4acf272b87476c30e4b47278bdac8fc1bc3ba7a7c434bdd9d839c96444779c23" exitCode=137
Nov 02 09:47:48.409373 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:48.409103    4449 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.203s"
Nov 02 09:47:48.409373 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.409189    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:47:48.436507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.435974    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:47:48.436507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.436064    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:47:48.436507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.436088    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:47:48.436507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.436109    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"]
Nov 02 09:47:48.448989 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.448768    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:47:48.529653 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.449886    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-f11c97"
Nov 02 09:47:48.541606 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:47:48.541566    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod767250c5_1f49_4bc9_b77f_0d2c8d55685e.slice/crio-4acf272b87476c30e4b47278bdac8fc1bc3ba7a7c434bdd9d839c96444779c23.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod767250c5_1f49_4bc9_b77f_0d2c8d55685e.slice/crio-conmon-4acf272b87476c30e4b47278bdac8fc1bc3ba7a7c434bdd9d839c96444779c23.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 09:47:48.626890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.624856    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10350da6-ad59-4afe-a58a-91ff5ae83f4a-kube-api-access-pxkcw\" (UniqueName: \"kubernetes.io/projected/10350da6-ad59-4afe-a58a-91ff5ae83f4a-kube-api-access-pxkcw\") pod \"10350da6-ad59-4afe-a58a-91ff5ae83f4a\" (UID: \"10350da6-ad59-4afe-a58a-91ff5ae83f4a\") "
Nov 02 09:47:48.626890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.625025    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:47:48.626890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.625080    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:48.636951 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.636912    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10350da6-ad59-4afe-a58a-91ff5ae83f4a-kube-api-access-pxkcw" pod "10350da6-ad59-4afe-a58a-91ff5ae83f4a" (UID: "10350da6-ad59-4afe-a58a-91ff5ae83f4a"). InnerVolumeSpecName "kube-api-access-pxkcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:48.732418 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.730520    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxkcw\" (UniqueName: \"kubernetes.io/projected/10350da6-ad59-4afe-a58a-91ff5ae83f4a-kube-api-access-pxkcw\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:48.774249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.755326    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-0c0970"
Nov 02 09:47:48.831899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.831866    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767250c5-1f49-4bc9-b77f-0d2c8d55685e-kube-api-access-tmnjr\" (UniqueName: \"kubernetes.io/projected/767250c5-1f49-4bc9-b77f-0d2c8d55685e-kube-api-access-tmnjr\") pod \"767250c5-1f49-4bc9-b77f-0d2c8d55685e\" (UID: \"767250c5-1f49-4bc9-b77f-0d2c8d55685e\") "
Nov 02 09:47:48.867593 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.866944    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:47:48.867593 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.867004    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" podUID="20132bdb-f0fa-4408-9a26-5b34c1510038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:48.924047 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.923966    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767250c5-1f49-4bc9-b77f-0d2c8d55685e-kube-api-access-tmnjr" pod "767250c5-1f49-4bc9-b77f-0d2c8d55685e" (UID: "767250c5-1f49-4bc9-b77f-0d2c8d55685e"). InnerVolumeSpecName "kube-api-access-tmnjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:48.936385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.936330    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmnjr\" (UniqueName: \"kubernetes.io/projected/767250c5-1f49-4bc9-b77f-0d2c8d55685e-kube-api-access-tmnjr\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:48.995428 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.995397    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:48.996371 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.996199    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:48.997833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:48.997778    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:47:49.093255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:49.093203    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:49.095114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:49.095087    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:49.099252 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:49.096808    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:47:49.208209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:49.208129    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10350da6-ad59-4afe-a58a-91ff5ae83f4a" path="/var/lib/kubelet/pods/10350da6-ad59-4afe-a58a-91ff5ae83f4a/volumes"
Nov 02 09:47:49.208562 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:49.208545    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767250c5-1f49-4bc9-b77f-0d2c8d55685e" path="/var/lib/kubelet/pods/767250c5-1f49-4bc9-b77f-0d2c8d55685e/volumes"
Nov 02 09:47:49.409807 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:49.409775    4449 scope.go:122] "RemoveContainer" containerID="130cc99d15e38e7a421120c19803e23c8ea6a054633af47d897351f042f94969"
Nov 02 09:47:49.409997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:49.409779    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-f11c97"
Nov 02 09:47:49.412026 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:49.411997    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" event={"ID":"f6a56df7-3e4e-421c-808b-c2fd2f440ea4","Type":"ContainerStarted","Data":"9549dcf35923b7245a92febf590db5190b9c6513ac282fbc1414b94f20e55793"}
Nov 02 09:47:49.412177 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:49.412041    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" event={"ID":"f6a56df7-3e4e-421c-808b-c2fd2f440ea4","Type":"ContainerStarted","Data":"42a4acae36ec9ada7a85a4ee682503c4d9b84da1086352520d65b1c68a657450"}
Nov 02 09:47:49.413920 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:49.413890    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-0c0970"
Nov 02 09:47:49.421049 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:49.421017    4449 scope.go:122] "RemoveContainer" containerID="4acf272b87476c30e4b47278bdac8fc1bc3ba7a7c434bdd9d839c96444779c23"
Nov 02 09:47:49.437486 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:49.437443    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podStartSLOduration=4.437431949 podStartE2EDuration="4.437431949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:49.433293725 +0000 UTC m=+746.371580529" watchObservedRunningTime="2026-11-02 09:47:49.437431949 +0000 UTC m=+746.375718733"
Nov 02 09:47:50.301319 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:50.301288    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:51.336027 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.335986    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"]
Nov 02 09:47:51.342110 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.342087    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"
Nov 02 09:47:51.346518 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.346492    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:47:51.346811 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.346607    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:47:51.347812 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.347788    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:47:51.347915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.347837    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:47:51.350694 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.350672    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"]
Nov 02 09:47:51.370111 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.370071    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54790839-bmgbr\" (UID: \"08edce8a-6a15-4062-83a4-977652f312c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"
Nov 02 09:47:51.370334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.370132    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54790839-bmgbr\" (UID: \"08edce8a-6a15-4062-83a4-977652f312c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"
Nov 02 09:47:51.370334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.370211    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-589rc\" (UniqueName: \"kubernetes.io/projected/08edce8a-6a15-4062-83a4-977652f312c8-kube-api-access-589rc\") pod \"maas-api-revoke-sub-54790839-bmgbr\" (UID: \"08edce8a-6a15-4062-83a4-977652f312c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"
Nov 02 09:47:51.471360 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.471324    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54790839-bmgbr\" (UID: \"08edce8a-6a15-4062-83a4-977652f312c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"
Nov 02 09:47:51.471569 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.471399    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54790839-bmgbr\" (UID: \"08edce8a-6a15-4062-83a4-977652f312c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"
Nov 02 09:47:51.471569 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.471447    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-589rc\" (UniqueName: \"kubernetes.io/projected/08edce8a-6a15-4062-83a4-977652f312c8-kube-api-access-589rc\") pod \"maas-api-revoke-sub-54790839-bmgbr\" (UID: \"08edce8a-6a15-4062-83a4-977652f312c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"
Nov 02 09:47:51.471989 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.471962    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54790839-bmgbr\" (UID: \"08edce8a-6a15-4062-83a4-977652f312c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"
Nov 02 09:47:51.474036 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.474010    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54790839-bmgbr\" (UID: \"08edce8a-6a15-4062-83a4-977652f312c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"
Nov 02 09:47:51.484514 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.484488    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-589rc\" (UniqueName: \"kubernetes.io/projected/08edce8a-6a15-4062-83a4-977652f312c8-kube-api-access-589rc\") pod \"maas-api-revoke-sub-54790839-bmgbr\" (UID: \"08edce8a-6a15-4062-83a4-977652f312c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"
Nov 02 09:47:51.660731 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.660641    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"
Nov 02 09:47:51.942282 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:51.942249    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"]
Nov 02 09:47:51.943134 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:47:51.943105    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08edce8a_6a15_4062_83a4_977652f312c8.slice/crio-54bcef06e457243ea79a7426f6adf1240702412fe131baf6e9a41843b041fe57 WatchSource:0}: Error finding container 54bcef06e457243ea79a7426f6adf1240702412fe131baf6e9a41843b041fe57: Status 404 returned error can't find the container with id 54bcef06e457243ea79a7426f6adf1240702412fe131baf6e9a41843b041fe57
Nov 02 09:47:52.443126 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:52.443090    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr" event={"ID":"08edce8a-6a15-4062-83a4-977652f312c8","Type":"ContainerStarted","Data":"0537ea13a742ef03a0f05a79faf2a4a0eb076ea2f87d30372cd623eafd769fd4"}
Nov 02 09:47:52.443622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:52.443159    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr" event={"ID":"08edce8a-6a15-4062-83a4-977652f312c8","Type":"ContainerStarted","Data":"54bcef06e457243ea79a7426f6adf1240702412fe131baf6e9a41843b041fe57"}
Nov 02 09:47:52.466397 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:52.466348    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr" podStartSLOduration=1.466333298 podStartE2EDuration="1.466333298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:52.461799232 +0000 UTC m=+749.400086009" watchObservedRunningTime="2026-11-02 09:47:52.466333298 +0000 UTC m=+749.404620073"
Nov 02 09:47:53.452338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:53.452299    4449 generic.go:356] "Generic (PLEG): container finished" podID="08edce8a-6a15-4062-83a4-977652f312c8" containerID="0537ea13a742ef03a0f05a79faf2a4a0eb076ea2f87d30372cd623eafd769fd4" exitCode=0
Nov 02 09:47:53.452741 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:53.452349    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr" event={"ID":"08edce8a-6a15-4062-83a4-977652f312c8","Type":"ContainerDied","Data":"0537ea13a742ef03a0f05a79faf2a4a0eb076ea2f87d30372cd623eafd769fd4"}
Nov 02 09:47:54.734761 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:54.734728    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"
Nov 02 09:47:54.837242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:54.829155    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08edce8a-6a15-4062-83a4-977652f312c8-kube-api-access-589rc\" (UniqueName: \"kubernetes.io/projected/08edce8a-6a15-4062-83a4-977652f312c8-kube-api-access-589rc\") pod \"08edce8a-6a15-4062-83a4-977652f312c8\" (UID: \"08edce8a-6a15-4062-83a4-977652f312c8\") "
Nov 02 09:47:54.837242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:54.829206    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-openshift-ca\") pod \"08edce8a-6a15-4062-83a4-977652f312c8\" (UID: \"08edce8a-6a15-4062-83a4-977652f312c8\") "
Nov 02 09:47:54.837242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:54.829404    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-xks-ca\") pod \"08edce8a-6a15-4062-83a4-977652f312c8\" (UID: \"08edce8a-6a15-4062-83a4-977652f312c8\") "
Nov 02 09:47:54.837242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:54.834005    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08edce8a-6a15-4062-83a4-977652f312c8-kube-api-access-589rc" pod "08edce8a-6a15-4062-83a4-977652f312c8" (UID: "08edce8a-6a15-4062-83a4-977652f312c8"). InnerVolumeSpecName "kube-api-access-589rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:54.837242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:54.834433    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-openshift-ca" pod "08edce8a-6a15-4062-83a4-977652f312c8" (UID: "08edce8a-6a15-4062-83a4-977652f312c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:47:54.865612 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:54.865554    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-xks-ca" pod "08edce8a-6a15-4062-83a4-977652f312c8" (UID: "08edce8a-6a15-4062-83a4-977652f312c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:47:54.942038 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:54.941999    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:54.942038 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:54.942037    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-589rc\" (UniqueName: \"kubernetes.io/projected/08edce8a-6a15-4062-83a4-977652f312c8-kube-api-access-589rc\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:54.942368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:54.942059    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08edce8a-6a15-4062-83a4-977652f312c8-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:55.500242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:55.496750    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr" event={"ID":"08edce8a-6a15-4062-83a4-977652f312c8","Type":"ContainerDied","Data":"54bcef06e457243ea79a7426f6adf1240702412fe131baf6e9a41843b041fe57"}
Nov 02 09:47:55.500242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:55.496813    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54bcef06e457243ea79a7426f6adf1240702412fe131baf6e9a41843b041fe57"
Nov 02 09:47:55.500242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:55.496883    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"
Nov 02 09:47:56.303244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:56.302991    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:56.795246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:56.789542    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:47:56.868560 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:47:56.866169    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:48:02.104046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.104016    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"]
Nov 02 09:48:02.104705 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.104690    4449 cpu_manager.go:414] "containerMap: removing container" podUID="08edce8a-6a15-4062-83a4-977652f312c8" containerName="revoke-keys"
Nov 02 09:48:02.104746 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.104706    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08edce8a-6a15-4062-83a4-977652f312c8" containerName="revoke-keys"
Nov 02 09:48:02.104814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.104805    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="08edce8a-6a15-4062-83a4-977652f312c8" containerName="revoke-keys"
Nov 02 09:48:02.109474 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.109458    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"
Nov 02 09:48:02.113587 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.113564    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:48:02.114441 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.114415    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:48:02.117177 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.117152    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"]
Nov 02 09:48:02.117344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.117327    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:48:02.117410 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.117389    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:48:02.279476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.279443    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg6dk\" (UniqueName: \"kubernetes.io/projected/9e355051-bae8-4cfd-bf5a-ab816171cce1-kube-api-access-hg6dk\") pod \"maas-api-revoke-sub-069ae59f-68q25\" (UID: \"9e355051-bae8-4cfd-bf5a-ab816171cce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"
Nov 02 09:48:02.279626 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.279508    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-069ae59f-68q25\" (UID: \"9e355051-bae8-4cfd-bf5a-ab816171cce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"
Nov 02 09:48:02.279680 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.279629    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-069ae59f-68q25\" (UID: \"9e355051-bae8-4cfd-bf5a-ab816171cce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"
Nov 02 09:48:02.381629 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.381551    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg6dk\" (UniqueName: \"kubernetes.io/projected/9e355051-bae8-4cfd-bf5a-ab816171cce1-kube-api-access-hg6dk\") pod \"maas-api-revoke-sub-069ae59f-68q25\" (UID: \"9e355051-bae8-4cfd-bf5a-ab816171cce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"
Nov 02 09:48:02.381629 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.381612    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-069ae59f-68q25\" (UID: \"9e355051-bae8-4cfd-bf5a-ab816171cce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"
Nov 02 09:48:02.381802 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.381636    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-069ae59f-68q25\" (UID: \"9e355051-bae8-4cfd-bf5a-ab816171cce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"
Nov 02 09:48:02.382285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.382258    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-069ae59f-68q25\" (UID: \"9e355051-bae8-4cfd-bf5a-ab816171cce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"
Nov 02 09:48:02.384308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.384287    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-069ae59f-68q25\" (UID: \"9e355051-bae8-4cfd-bf5a-ab816171cce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"
Nov 02 09:48:02.394043 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.394018    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg6dk\" (UniqueName: \"kubernetes.io/projected/9e355051-bae8-4cfd-bf5a-ab816171cce1-kube-api-access-hg6dk\") pod \"maas-api-revoke-sub-069ae59f-68q25\" (UID: \"9e355051-bae8-4cfd-bf5a-ab816171cce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"
Nov 02 09:48:02.427506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.427475    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"
Nov 02 09:48:02.733909 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:02.733876    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"]
Nov 02 09:48:02.734099 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:48:02.734074    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e355051_bae8_4cfd_bf5a_ab816171cce1.slice/crio-b60799c29420819a2e37a93c1d5ac7468141f438b58c02a908f74b6d0921a27e WatchSource:0}: Error finding container b60799c29420819a2e37a93c1d5ac7468141f438b58c02a908f74b6d0921a27e: Status 404 returned error can't find the container with id b60799c29420819a2e37a93c1d5ac7468141f438b58c02a908f74b6d0921a27e
Nov 02 09:48:03.582135 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.582095    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25" event={"ID":"9e355051-bae8-4cfd-bf5a-ab816171cce1","Type":"ContainerStarted","Data":"5dfedac38e23f8b0aabfde50038331ae4e45261acfbd40b82a704f2ce2144ac0"}
Nov 02 09:48:03.582544 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.582163    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25" event={"ID":"9e355051-bae8-4cfd-bf5a-ab816171cce1","Type":"ContainerStarted","Data":"b60799c29420819a2e37a93c1d5ac7468141f438b58c02a908f74b6d0921a27e"}
Nov 02 09:48:03.601619 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.601566    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25" podStartSLOduration=1.601554465 podStartE2EDuration="1.601554465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:48:03.599916192 +0000 UTC m=+760.538202996" watchObservedRunningTime="2026-11-02 09:48:03.601554465 +0000 UTC m=+760.539841240"
Nov 02 09:48:03.643067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.643037    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"]
Nov 02 09:48:03.648922 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.648896    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.657483 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.657402    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9e7156\"/\"openshift-service-ca.crt\""
Nov 02 09:48:03.658576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.658429    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9e7156\"/\"rate-test-model-9e7156-kserve-self-signed-certs\""
Nov 02 09:48:03.658576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.658458    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9e7156\"/\"kube-root-ca.crt\""
Nov 02 09:48:03.658576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.658554    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9e7156\"/\"default-dockercfg-8h4tz\""
Nov 02 09:48:03.661270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.661246    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"]
Nov 02 09:48:03.698603 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.698578    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-dshm\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.698744 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.698650    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-model-cache\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.698818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.698797    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tls-certs\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.698881 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.698847    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w24d\" (UniqueName: \"kubernetes.io/projected/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-kube-api-access-6w24d\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.699018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.698999    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-home\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.699081 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.699052    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tmp-dir\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.800231 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.800195    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-model-cache\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.800336 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.800319    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tls-certs\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.800393 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.800367    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w24d\" (UniqueName: \"kubernetes.io/projected/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-kube-api-access-6w24d\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.800480 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.800463    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-home\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.801114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.800520    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tmp-dir\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.801114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.800597    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-model-cache\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.801114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.800624    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-dshm\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.801114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.800910    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tmp-dir\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.801114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.800928    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-home\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.802954 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.802931    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-dshm\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.803047 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.802940    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tls-certs\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.812675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.812656    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w24d\" (UniqueName: \"kubernetes.io/projected/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-kube-api-access-6w24d\") pod \"rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:03.972700 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:03.972369    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:04.281801 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:48:04.281774    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72db0ef3_c161_4bf2_b844_a8aabb71c8bc.slice/crio-30b0410af32094804abf45425c1c5ac2ede6cca95151ccad15ffd68557180851 WatchSource:0}: Error finding container 30b0410af32094804abf45425c1c5ac2ede6cca95151ccad15ffd68557180851: Status 404 returned error can't find the container with id 30b0410af32094804abf45425c1c5ac2ede6cca95151ccad15ffd68557180851
Nov 02 09:48:04.282988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:04.282962    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"]
Nov 02 09:48:04.595888 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:04.595856    4449 generic.go:356] "Generic (PLEG): container finished" podID="9e355051-bae8-4cfd-bf5a-ab816171cce1" containerID="5dfedac38e23f8b0aabfde50038331ae4e45261acfbd40b82a704f2ce2144ac0" exitCode=0
Nov 02 09:48:04.596386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:04.595941    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25" event={"ID":"9e355051-bae8-4cfd-bf5a-ab816171cce1","Type":"ContainerDied","Data":"5dfedac38e23f8b0aabfde50038331ae4e45261acfbd40b82a704f2ce2144ac0"}
Nov 02 09:48:04.598272 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:04.598184    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" event={"ID":"72db0ef3-c161-4bf2-b844-a8aabb71c8bc","Type":"ContainerStarted","Data":"4034954677b3d4bbc4a02e1a6648868af634025219ee0c74c1060c1a1cecf137"}
Nov 02 09:48:04.598414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:04.598275    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" event={"ID":"72db0ef3-c161-4bf2-b844-a8aabb71c8bc","Type":"ContainerStarted","Data":"30b0410af32094804abf45425c1c5ac2ede6cca95151ccad15ffd68557180851"}
Nov 02 09:48:04.638914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:04.638867    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podStartSLOduration=1.6388550419999999 podStartE2EDuration="1.638855042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:48:04.635616127 +0000 UTC m=+761.573902903" watchObservedRunningTime="2026-11-02 09:48:04.638855042 +0000 UTC m=+761.577141817"
Nov 02 09:48:04.973100 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:04.973018    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:05.768161 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.768136    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"
Nov 02 09:48:05.832753 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.832722    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-openshift-ca\") pod \"9e355051-bae8-4cfd-bf5a-ab816171cce1\" (UID: \"9e355051-bae8-4cfd-bf5a-ab816171cce1\") "
Nov 02 09:48:05.833117 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.833069    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-openshift-ca" pod "9e355051-bae8-4cfd-bf5a-ab816171cce1" (UID: "9e355051-bae8-4cfd-bf5a-ab816171cce1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:48:05.833117 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.833071    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e355051-bae8-4cfd-bf5a-ab816171cce1-kube-api-access-hg6dk\" (UniqueName: \"kubernetes.io/projected/9e355051-bae8-4cfd-bf5a-ab816171cce1-kube-api-access-hg6dk\") pod \"9e355051-bae8-4cfd-bf5a-ab816171cce1\" (UID: \"9e355051-bae8-4cfd-bf5a-ab816171cce1\") "
Nov 02 09:48:05.834399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.833178    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-xks-ca\") pod \"9e355051-bae8-4cfd-bf5a-ab816171cce1\" (UID: \"9e355051-bae8-4cfd-bf5a-ab816171cce1\") "
Nov 02 09:48:05.834399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.833841    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:05.841561 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.841521    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e355051-bae8-4cfd-bf5a-ab816171cce1-kube-api-access-hg6dk" pod "9e355051-bae8-4cfd-bf5a-ab816171cce1" (UID: "9e355051-bae8-4cfd-bf5a-ab816171cce1"). InnerVolumeSpecName "kube-api-access-hg6dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:48:05.842668 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.842639    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-xks-ca" pod "9e355051-bae8-4cfd-bf5a-ab816171cce1" (UID: "9e355051-bae8-4cfd-bf5a-ab816171cce1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:48:05.935492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.935426    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg6dk\" (UniqueName: \"kubernetes.io/projected/9e355051-bae8-4cfd-bf5a-ab816171cce1-kube-api-access-hg6dk\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:05.935492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.935452    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e355051-bae8-4cfd-bf5a-ab816171cce1-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:05.982726 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.982694    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"]
Nov 02 09:48:05.983381 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.983366    4449 cpu_manager.go:414] "containerMap: removing container" podUID="9e355051-bae8-4cfd-bf5a-ab816171cce1" containerName="revoke-keys"
Nov 02 09:48:05.983420 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.983383    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e355051-bae8-4cfd-bf5a-ab816171cce1" containerName="revoke-keys"
Nov 02 09:48:05.983541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.983529    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e355051-bae8-4cfd-bf5a-ab816171cce1" containerName="revoke-keys"
Nov 02 09:48:05.987300 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.987276    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:05.993200 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.993167    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-plugins-e2e-extmod-b-204fba\""
Nov 02 09:48:05.993200 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.993190    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"openshift-service-ca.crt\""
Nov 02 09:48:05.993387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.993271    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"kube-root-ca.crt\""
Nov 02 09:48:05.993600 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.993579    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"routing-overlay\""
Nov 02 09:48:05.998783 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:05.998759    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"]
Nov 02 09:48:06.027757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.027727    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-1521404c1537\""
Nov 02 09:48:06.027757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.027737    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-e2e--e7669bbb-dockercfg-zngpv\""
Nov 02 09:48:06.036895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.036857    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dmrz\" (UniqueName: \"kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-kube-api-access-6dmrz\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:06.037135 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.037108    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:06.037254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.037231    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:06.037323 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.037294    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:06.142506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.140083    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:06.142506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.140240    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:06.142506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.140304    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:06.142506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.140376    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dmrz\" (UniqueName: \"kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-kube-api-access-6dmrz\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:06.142506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.141932    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:06.146052 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.146020    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:06.190289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.182162    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:06.197011 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.196976    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dmrz\" (UniqueName: \"kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-kube-api-access-6dmrz\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:06.304678 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.304639    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:06.552538 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:48:06.552495    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:48:06.608623 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:48:06.608595    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b927fd_89fb_40ba_bf6c_b4ca5d926ab7.slice/crio-3519ae36dfe3bf60107d9d994dae2ba78a99f246f2dd8e694c2e4d7d51784b26 WatchSource:0}: Error finding container 3519ae36dfe3bf60107d9d994dae2ba78a99f246f2dd8e694c2e4d7d51784b26: Status 404 returned error can't find the container with id 3519ae36dfe3bf60107d9d994dae2ba78a99f246f2dd8e694c2e4d7d51784b26
Nov 02 09:48:06.609320 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.608940    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"]
Nov 02 09:48:06.617615 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.617588    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"
Nov 02 09:48:06.618350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.618310    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25" event={"ID":"9e355051-bae8-4cfd-bf5a-ab816171cce1","Type":"ContainerDied","Data":"b60799c29420819a2e37a93c1d5ac7468141f438b58c02a908f74b6d0921a27e"}
Nov 02 09:48:06.618503 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:06.618380    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b60799c29420819a2e37a93c1d5ac7468141f438b58c02a908f74b6d0921a27e"
Nov 02 09:48:07.629435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:07.629387    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp" event={"ID":"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7","Type":"ContainerStarted","Data":"f31ca3a1439e6f63c1d95ad7e68dbc8e6b1d236059284465cead3c416b83976b"}
Nov 02 09:48:07.629956 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:07.629464    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp" event={"ID":"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7","Type":"ContainerStarted","Data":"3519ae36dfe3bf60107d9d994dae2ba78a99f246f2dd8e694c2e4d7d51784b26"}
Nov 02 09:48:07.629956 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:07.629498    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:07.651689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:07.651632    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp" podStartSLOduration=2.651618964 podStartE2EDuration="2.651618964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:48:07.648606805 +0000 UTC m=+764.586893604" watchObservedRunningTime="2026-11-02 09:48:07.651618964 +0000 UTC m=+764.589905740"
Nov 02 09:48:10.275603 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.275560    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"]
Nov 02 09:48:10.280464 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.280432    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"
Nov 02 09:48:10.285605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.285568    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:48:10.285764 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.285623    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:48:10.285764 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.285663    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:48:10.286367 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.286339    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:48:10.301714 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.291513    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"]
Nov 02 09:48:10.358722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.358684    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3084fb1d-7qcnl\" (UID: \"f5978d00-1798-4178-8ffd-7c3c76150f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"
Nov 02 09:48:10.358908 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.358883    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3084fb1d-7qcnl\" (UID: \"f5978d00-1798-4178-8ffd-7c3c76150f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"
Nov 02 09:48:10.358959 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.358935    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vnlv\" (UniqueName: \"kubernetes.io/projected/f5978d00-1798-4178-8ffd-7c3c76150f2f-kube-api-access-4vnlv\") pod \"maas-api-revoke-sub-3084fb1d-7qcnl\" (UID: \"f5978d00-1798-4178-8ffd-7c3c76150f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"
Nov 02 09:48:10.460545 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.460514    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3084fb1d-7qcnl\" (UID: \"f5978d00-1798-4178-8ffd-7c3c76150f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"
Nov 02 09:48:10.460725 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.460576    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vnlv\" (UniqueName: \"kubernetes.io/projected/f5978d00-1798-4178-8ffd-7c3c76150f2f-kube-api-access-4vnlv\") pod \"maas-api-revoke-sub-3084fb1d-7qcnl\" (UID: \"f5978d00-1798-4178-8ffd-7c3c76150f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"
Nov 02 09:48:10.460725 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.460670    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3084fb1d-7qcnl\" (UID: \"f5978d00-1798-4178-8ffd-7c3c76150f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"
Nov 02 09:48:10.461391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.461287    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3084fb1d-7qcnl\" (UID: \"f5978d00-1798-4178-8ffd-7c3c76150f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"
Nov 02 09:48:10.462997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.462972    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3084fb1d-7qcnl\" (UID: \"f5978d00-1798-4178-8ffd-7c3c76150f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"
Nov 02 09:48:10.473065 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.473035    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vnlv\" (UniqueName: \"kubernetes.io/projected/f5978d00-1798-4178-8ffd-7c3c76150f2f-kube-api-access-4vnlv\") pod \"maas-api-revoke-sub-3084fb1d-7qcnl\" (UID: \"f5978d00-1798-4178-8ffd-7c3c76150f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"
Nov 02 09:48:10.601675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.601639    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"
Nov 02 09:48:10.897411 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:48:10.897381    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5978d00_1798_4178_8ffd_7c3c76150f2f.slice/crio-eb63a1199726bf46e7fec5590d9a71cad5041362d0cffa7a1af28547598b3d97 WatchSource:0}: Error finding container eb63a1199726bf46e7fec5590d9a71cad5041362d0cffa7a1af28547598b3d97: Status 404 returned error can't find the container with id eb63a1199726bf46e7fec5590d9a71cad5041362d0cffa7a1af28547598b3d97
Nov 02 09:48:10.897623 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:10.897602    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"]
Nov 02 09:48:11.660582 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:11.660544    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl" event={"ID":"f5978d00-1798-4178-8ffd-7c3c76150f2f","Type":"ContainerStarted","Data":"32c1498603deeafe49fd9c6b473b8da22b8babc3e505bc57a02a9b0fd2f49b3c"}
Nov 02 09:48:11.660989 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:11.660598    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl" event={"ID":"f5978d00-1798-4178-8ffd-7c3c76150f2f","Type":"ContainerStarted","Data":"eb63a1199726bf46e7fec5590d9a71cad5041362d0cffa7a1af28547598b3d97"}
Nov 02 09:48:11.686119 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:11.686072    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl" podStartSLOduration=1.686058198 podStartE2EDuration="1.686058198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:48:11.681894806 +0000 UTC m=+768.620181593" watchObservedRunningTime="2026-11-02 09:48:11.686058198 +0000 UTC m=+768.624344974"
Nov 02 09:48:12.670177 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:12.670137    4449 generic.go:356] "Generic (PLEG): container finished" podID="f5978d00-1798-4178-8ffd-7c3c76150f2f" containerID="32c1498603deeafe49fd9c6b473b8da22b8babc3e505bc57a02a9b0fd2f49b3c" exitCode=0
Nov 02 09:48:12.670675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:12.670236    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl" event={"ID":"f5978d00-1798-4178-8ffd-7c3c76150f2f","Type":"ContainerDied","Data":"32c1498603deeafe49fd9c6b473b8da22b8babc3e505bc57a02a9b0fd2f49b3c"}
Nov 02 09:48:13.855825 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:13.855796    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"
Nov 02 09:48:13.907439 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:13.907408    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-xks-ca\") pod \"f5978d00-1798-4178-8ffd-7c3c76150f2f\" (UID: \"f5978d00-1798-4178-8ffd-7c3c76150f2f\") "
Nov 02 09:48:13.907621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:13.907467    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5978d00-1798-4178-8ffd-7c3c76150f2f-kube-api-access-4vnlv\" (UniqueName: \"kubernetes.io/projected/f5978d00-1798-4178-8ffd-7c3c76150f2f-kube-api-access-4vnlv\") pod \"f5978d00-1798-4178-8ffd-7c3c76150f2f\" (UID: \"f5978d00-1798-4178-8ffd-7c3c76150f2f\") "
Nov 02 09:48:13.907621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:13.907512    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-openshift-ca\") pod \"f5978d00-1798-4178-8ffd-7c3c76150f2f\" (UID: \"f5978d00-1798-4178-8ffd-7c3c76150f2f\") "
Nov 02 09:48:13.907879 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:13.907852    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-openshift-ca" pod "f5978d00-1798-4178-8ffd-7c3c76150f2f" (UID: "f5978d00-1798-4178-8ffd-7c3c76150f2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:48:13.908061 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:13.908041    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:13.909721 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:13.909687    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-xks-ca" pod "f5978d00-1798-4178-8ffd-7c3c76150f2f" (UID: "f5978d00-1798-4178-8ffd-7c3c76150f2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:48:13.909967 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:13.909942    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5978d00-1798-4178-8ffd-7c3c76150f2f-kube-api-access-4vnlv" pod "f5978d00-1798-4178-8ffd-7c3c76150f2f" (UID: "f5978d00-1798-4178-8ffd-7c3c76150f2f"). InnerVolumeSpecName "kube-api-access-4vnlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:48:13.973516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:13.973416    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:13.985781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:13.985758    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:14.009313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.009283    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5978d00-1798-4178-8ffd-7c3c76150f2f-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:14.009313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.009310    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vnlv\" (UniqueName: \"kubernetes.io/projected/f5978d00-1798-4178-8ffd-7c3c76150f2f-kube-api-access-4vnlv\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:14.644470 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.644432    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"]
Nov 02 09:48:14.645105 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.645086    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f5978d00-1798-4178-8ffd-7c3c76150f2f" containerName="revoke-keys"
Nov 02 09:48:14.645105 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.645101    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5978d00-1798-4178-8ffd-7c3c76150f2f" containerName="revoke-keys"
Nov 02 09:48:14.645318 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.645194    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5978d00-1798-4178-8ffd-7c3c76150f2f" containerName="revoke-keys"
Nov 02 09:48:14.649013 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.648995    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"
Nov 02 09:48:14.656255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.656197    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"]
Nov 02 09:48:14.697966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.697927    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"
Nov 02 09:48:14.698158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.697937    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl" event={"ID":"f5978d00-1798-4178-8ffd-7c3c76150f2f","Type":"ContainerDied","Data":"eb63a1199726bf46e7fec5590d9a71cad5041362d0cffa7a1af28547598b3d97"}
Nov 02 09:48:14.698158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.698053    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb63a1199726bf46e7fec5590d9a71cad5041362d0cffa7a1af28547598b3d97"
Nov 02 09:48:14.715376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.715353    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:48:14.719143 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.719120    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc6d3801-rccb5\" (UID: \"3e7ff987-a493-412b-a48c-c53c17e61823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"
Nov 02 09:48:14.719393 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.719373    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sttxp\" (UniqueName: \"kubernetes.io/projected/3e7ff987-a493-412b-a48c-c53c17e61823-kube-api-access-sttxp\") pod \"maas-api-revoke-sub-cc6d3801-rccb5\" (UID: \"3e7ff987-a493-412b-a48c-c53c17e61823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"
Nov 02 09:48:14.719455 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.719443    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc6d3801-rccb5\" (UID: \"3e7ff987-a493-412b-a48c-c53c17e61823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"
Nov 02 09:48:14.821332 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.821303    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sttxp\" (UniqueName: \"kubernetes.io/projected/3e7ff987-a493-412b-a48c-c53c17e61823-kube-api-access-sttxp\") pod \"maas-api-revoke-sub-cc6d3801-rccb5\" (UID: \"3e7ff987-a493-412b-a48c-c53c17e61823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"
Nov 02 09:48:14.821511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.821364    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc6d3801-rccb5\" (UID: \"3e7ff987-a493-412b-a48c-c53c17e61823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"
Nov 02 09:48:14.821511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.821435    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc6d3801-rccb5\" (UID: \"3e7ff987-a493-412b-a48c-c53c17e61823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"
Nov 02 09:48:14.822128 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.822063    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc6d3801-rccb5\" (UID: \"3e7ff987-a493-412b-a48c-c53c17e61823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"
Nov 02 09:48:14.823908 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.823886    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc6d3801-rccb5\" (UID: \"3e7ff987-a493-412b-a48c-c53c17e61823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"
Nov 02 09:48:14.833879 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.833853    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sttxp\" (UniqueName: \"kubernetes.io/projected/3e7ff987-a493-412b-a48c-c53c17e61823-kube-api-access-sttxp\") pod \"maas-api-revoke-sub-cc6d3801-rccb5\" (UID: \"3e7ff987-a493-412b-a48c-c53c17e61823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"
Nov 02 09:48:14.967814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:14.967714    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"
Nov 02 09:48:15.263094 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:48:15.263062    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7ff987_a493_412b_a48c_c53c17e61823.slice/crio-de8e986c5bb6981062ee63e33771c03b3a94610f97551bd3dbb8f8264eec681b WatchSource:0}: Error finding container de8e986c5bb6981062ee63e33771c03b3a94610f97551bd3dbb8f8264eec681b: Status 404 returned error can't find the container with id de8e986c5bb6981062ee63e33771c03b3a94610f97551bd3dbb8f8264eec681b
Nov 02 09:48:15.265628 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:15.265599    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"]
Nov 02 09:48:15.722266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:15.722154    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5" event={"ID":"3e7ff987-a493-412b-a48c-c53c17e61823","Type":"ContainerStarted","Data":"49650b5045b4d70e0c6e80f46592299b63371eb3dbd2e626fe748bc4d7b0f86e"}
Nov 02 09:48:15.722266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:15.722209    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5" event={"ID":"3e7ff987-a493-412b-a48c-c53c17e61823","Type":"ContainerStarted","Data":"de8e986c5bb6981062ee63e33771c03b3a94610f97551bd3dbb8f8264eec681b"}
Nov 02 09:48:15.755238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:15.751706    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5" podStartSLOduration=1.751686237 podStartE2EDuration="1.751686237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:48:15.748995003 +0000 UTC m=+772.687281796" watchObservedRunningTime="2026-11-02 09:48:15.751686237 +0000 UTC m=+772.689973015"
Nov 02 09:48:16.762244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:16.761132    4449 generic.go:356] "Generic (PLEG): container finished" podID="3e7ff987-a493-412b-a48c-c53c17e61823" containerID="49650b5045b4d70e0c6e80f46592299b63371eb3dbd2e626fe748bc4d7b0f86e" exitCode=0
Nov 02 09:48:16.762244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:16.761202    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5" event={"ID":"3e7ff987-a493-412b-a48c-c53c17e61823","Type":"ContainerDied","Data":"49650b5045b4d70e0c6e80f46592299b63371eb3dbd2e626fe748bc4d7b0f86e"}
Nov 02 09:48:18.427255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.427207    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"
Nov 02 09:48:18.532878 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.532131    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-openshift-ca\") pod \"3e7ff987-a493-412b-a48c-c53c17e61823\" (UID: \"3e7ff987-a493-412b-a48c-c53c17e61823\") "
Nov 02 09:48:18.532878 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.532182    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-xks-ca\") pod \"3e7ff987-a493-412b-a48c-c53c17e61823\" (UID: \"3e7ff987-a493-412b-a48c-c53c17e61823\") "
Nov 02 09:48:18.532878 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.532332    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7ff987-a493-412b-a48c-c53c17e61823-kube-api-access-sttxp\" (UniqueName: \"kubernetes.io/projected/3e7ff987-a493-412b-a48c-c53c17e61823-kube-api-access-sttxp\") pod \"3e7ff987-a493-412b-a48c-c53c17e61823\" (UID: \"3e7ff987-a493-412b-a48c-c53c17e61823\") "
Nov 02 09:48:18.532878 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.532597    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-openshift-ca" pod "3e7ff987-a493-412b-a48c-c53c17e61823" (UID: "3e7ff987-a493-412b-a48c-c53c17e61823"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:48:18.533175 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.532975    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:18.538418 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.538372    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7ff987-a493-412b-a48c-c53c17e61823-kube-api-access-sttxp" pod "3e7ff987-a493-412b-a48c-c53c17e61823" (UID: "3e7ff987-a493-412b-a48c-c53c17e61823"). InnerVolumeSpecName "kube-api-access-sttxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:48:18.544335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.539200    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-xks-ca" pod "3e7ff987-a493-412b-a48c-c53c17e61823" (UID: "3e7ff987-a493-412b-a48c-c53c17e61823"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:48:18.634533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.634442    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7ff987-a493-412b-a48c-c53c17e61823-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:18.634533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.634476    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sttxp\" (UniqueName: \"kubernetes.io/projected/3e7ff987-a493-412b-a48c-c53c17e61823-kube-api-access-sttxp\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:18.639564 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.639537    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:48:18.804444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.804389    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"
Nov 02 09:48:18.804444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.804402    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5" event={"ID":"3e7ff987-a493-412b-a48c-c53c17e61823","Type":"ContainerDied","Data":"de8e986c5bb6981062ee63e33771c03b3a94610f97551bd3dbb8f8264eec681b"}
Nov 02 09:48:18.804705 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:18.804466    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de8e986c5bb6981062ee63e33771c03b3a94610f97551bd3dbb8f8264eec681b"
Nov 02 09:48:22.230973 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.230940    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"]
Nov 02 09:48:22.231713 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.231693    4449 cpu_manager.go:414] "containerMap: removing container" podUID="3e7ff987-a493-412b-a48c-c53c17e61823" containerName="revoke-keys"
Nov 02 09:48:22.231765 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.231715    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e7ff987-a493-412b-a48c-c53c17e61823" containerName="revoke-keys"
Nov 02 09:48:22.231834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.231825    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e7ff987-a493-412b-a48c-c53c17e61823" containerName="revoke-keys"
Nov 02 09:48:22.235702 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.235684    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"
Nov 02 09:48:22.243838 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.243805    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:48:22.244008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.243902    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:48:22.244008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.243954    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:48:22.244143 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.244031    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:48:22.246571 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.246041    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"]
Nov 02 09:48:22.282192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.282147    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c35c15a1-mdcpw\" (UID: \"5f25a86c-09f9-4304-9570-f13da4338ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"
Nov 02 09:48:22.282443 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.282368    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c35c15a1-mdcpw\" (UID: \"5f25a86c-09f9-4304-9570-f13da4338ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"
Nov 02 09:48:22.282523 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.282440    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5znb\" (UniqueName: \"kubernetes.io/projected/5f25a86c-09f9-4304-9570-f13da4338ef3-kube-api-access-b5znb\") pod \"maas-api-revoke-sub-c35c15a1-mdcpw\" (UID: \"5f25a86c-09f9-4304-9570-f13da4338ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"
Nov 02 09:48:22.384290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.384193    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5znb\" (UniqueName: \"kubernetes.io/projected/5f25a86c-09f9-4304-9570-f13da4338ef3-kube-api-access-b5znb\") pod \"maas-api-revoke-sub-c35c15a1-mdcpw\" (UID: \"5f25a86c-09f9-4304-9570-f13da4338ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"
Nov 02 09:48:22.384514 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.384411    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c35c15a1-mdcpw\" (UID: \"5f25a86c-09f9-4304-9570-f13da4338ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"
Nov 02 09:48:22.384594 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.384534    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c35c15a1-mdcpw\" (UID: \"5f25a86c-09f9-4304-9570-f13da4338ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"
Nov 02 09:48:22.385151 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.385123    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c35c15a1-mdcpw\" (UID: \"5f25a86c-09f9-4304-9570-f13da4338ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"
Nov 02 09:48:22.387075 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.387046    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c35c15a1-mdcpw\" (UID: \"5f25a86c-09f9-4304-9570-f13da4338ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"
Nov 02 09:48:22.397671 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.397635    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5znb\" (UniqueName: \"kubernetes.io/projected/5f25a86c-09f9-4304-9570-f13da4338ef3-kube-api-access-b5znb\") pod \"maas-api-revoke-sub-c35c15a1-mdcpw\" (UID: \"5f25a86c-09f9-4304-9570-f13da4338ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"
Nov 02 09:48:22.555650 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.555600    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"
Nov 02 09:48:22.849607 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:22.849523    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"]
Nov 02 09:48:22.854390 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:48:22.854355    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f25a86c_09f9_4304_9570_f13da4338ef3.slice/crio-cedc36d37890557c44b2354ced03d74a87c4b3e17303743efe81d4e750dc1e69 WatchSource:0}: Error finding container cedc36d37890557c44b2354ced03d74a87c4b3e17303743efe81d4e750dc1e69: Status 404 returned error can't find the container with id cedc36d37890557c44b2354ced03d74a87c4b3e17303743efe81d4e750dc1e69
Nov 02 09:48:23.851621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:23.851580    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw" event={"ID":"5f25a86c-09f9-4304-9570-f13da4338ef3","Type":"ContainerStarted","Data":"96d5f93cab8251bbbf47aef9563102abc1a3a661e81b834f5b921bbe6bd79307"}
Nov 02 09:48:23.852256 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:23.852207    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw" event={"ID":"5f25a86c-09f9-4304-9570-f13da4338ef3","Type":"ContainerStarted","Data":"cedc36d37890557c44b2354ced03d74a87c4b3e17303743efe81d4e750dc1e69"}
Nov 02 09:48:23.926251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:23.908619    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw" podStartSLOduration=1.9085997620000001 podStartE2EDuration="1.908599762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:48:23.884770486 +0000 UTC m=+780.823057290" watchObservedRunningTime="2026-11-02 09:48:23.908599762 +0000 UTC m=+780.846886540"
Nov 02 09:48:24.861483 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:24.861443    4449 generic.go:356] "Generic (PLEG): container finished" podID="5f25a86c-09f9-4304-9570-f13da4338ef3" containerID="96d5f93cab8251bbbf47aef9563102abc1a3a661e81b834f5b921bbe6bd79307" exitCode=0
Nov 02 09:48:24.861891 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:24.861523    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw" event={"ID":"5f25a86c-09f9-4304-9570-f13da4338ef3","Type":"ContainerDied","Data":"96d5f93cab8251bbbf47aef9563102abc1a3a661e81b834f5b921bbe6bd79307"}
Nov 02 09:48:26.045422 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.045393    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"
Nov 02 09:48:26.062987 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.062947    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f25a86c-09f9-4304-9570-f13da4338ef3-kube-api-access-b5znb\" (UniqueName: \"kubernetes.io/projected/5f25a86c-09f9-4304-9570-f13da4338ef3-kube-api-access-b5znb\") pod \"5f25a86c-09f9-4304-9570-f13da4338ef3\" (UID: \"5f25a86c-09f9-4304-9570-f13da4338ef3\") "
Nov 02 09:48:26.063153 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.063095    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-openshift-ca\") pod \"5f25a86c-09f9-4304-9570-f13da4338ef3\" (UID: \"5f25a86c-09f9-4304-9570-f13da4338ef3\") "
Nov 02 09:48:26.064024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.063792    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-xks-ca\") pod \"5f25a86c-09f9-4304-9570-f13da4338ef3\" (UID: \"5f25a86c-09f9-4304-9570-f13da4338ef3\") "
Nov 02 09:48:26.064187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.064093    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-openshift-ca" pod "5f25a86c-09f9-4304-9570-f13da4338ef3" (UID: "5f25a86c-09f9-4304-9570-f13da4338ef3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:48:26.064701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.064675    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:26.066866 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.066822    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f25a86c-09f9-4304-9570-f13da4338ef3-kube-api-access-b5znb" pod "5f25a86c-09f9-4304-9570-f13da4338ef3" (UID: "5f25a86c-09f9-4304-9570-f13da4338ef3"). InnerVolumeSpecName "kube-api-access-b5znb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:48:26.066984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.066961    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-xks-ca" pod "5f25a86c-09f9-4304-9570-f13da4338ef3" (UID: "5f25a86c-09f9-4304-9570-f13da4338ef3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:48:26.165610 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.165507    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f25a86c-09f9-4304-9570-f13da4338ef3-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:26.165610 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.165543    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5znb\" (UniqueName: \"kubernetes.io/projected/5f25a86c-09f9-4304-9570-f13da4338ef3-kube-api-access-b5znb\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:26.370933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.370899    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"]
Nov 02 09:48:26.371814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.371786    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5f25a86c-09f9-4304-9570-f13da4338ef3" containerName="revoke-keys"
Nov 02 09:48:26.371814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.371810    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f25a86c-09f9-4304-9570-f13da4338ef3" containerName="revoke-keys"
Nov 02 09:48:26.372078 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.372060    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f25a86c-09f9-4304-9570-f13da4338ef3" containerName="revoke-keys"
Nov 02 09:48:26.376117 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.376089    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.395502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.395471    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"]
Nov 02 09:48:26.470334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.470229    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5071cf6-026d-407b-84b3-f682c15a8d22-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.470334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.470301    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.470538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.470434    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.470538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.470516    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.470614 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.470595    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.470687 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.470667    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg88d\" (UniqueName: \"kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-kube-api-access-cg88d\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.470730 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.470705    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.470782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.470730    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.470834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.470780    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.572582 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.572530    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5071cf6-026d-407b-84b3-f682c15a8d22-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.572796 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.572605    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.572796 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.572671    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.572796 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.572698    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.572796 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.572759    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.573194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.572804    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg88d\" (UniqueName: \"kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-kube-api-access-cg88d\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.573194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.572840    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.573194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.572881    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.573194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.572910    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.573470 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.573276    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.573470 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.573339    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.573470 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.573366    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.573586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.573516    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.573685 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.573539    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5071cf6-026d-407b-84b3-f682c15a8d22-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.575331 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.575302    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.575758 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.575734    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.585665 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.585633    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.585910 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.585893    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg88d\" (UniqueName: \"kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-kube-api-access-cg88d\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.703530 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.703487    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:26.879547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.879517    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"
Nov 02 09:48:26.879683 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.879522    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw" event={"ID":"5f25a86c-09f9-4304-9570-f13da4338ef3","Type":"ContainerDied","Data":"cedc36d37890557c44b2354ced03d74a87c4b3e17303743efe81d4e750dc1e69"}
Nov 02 09:48:26.879683 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.879636    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cedc36d37890557c44b2354ced03d74a87c4b3e17303743efe81d4e750dc1e69"
Nov 02 09:48:26.996535 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:48:26.996493    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5071cf6_026d_407b_84b3_f682c15a8d22.slice/crio-53ab7a7d8875b4e11fb7598fb7d94806455a4be6b60eb0883563b2440e4ca6da WatchSource:0}: Error finding container 53ab7a7d8875b4e11fb7598fb7d94806455a4be6b60eb0883563b2440e4ca6da: Status 404 returned error can't find the container with id 53ab7a7d8875b4e11fb7598fb7d94806455a4be6b60eb0883563b2440e4ca6da
Nov 02 09:48:26.997622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.997598    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"]
Nov 02 09:48:26.999337 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.999301    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:48:26.999449 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.999375    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:48:26.999449 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:26.999424    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:48:27.899785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:27.899745    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm" event={"ID":"f5071cf6-026d-407b-84b3-f682c15a8d22","Type":"ContainerStarted","Data":"19c2cba41a689ece661d978de644dc68f5e8763df72fb8f2e0381df12911d14d"}
Nov 02 09:48:27.900282 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:27.899817    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm" event={"ID":"f5071cf6-026d-407b-84b3-f682c15a8d22","Type":"ContainerStarted","Data":"53ab7a7d8875b4e11fb7598fb7d94806455a4be6b60eb0883563b2440e4ca6da"}
Nov 02 09:48:27.920317 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:27.920282    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"]
Nov 02 09:48:27.924475 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:27.924423    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm" podStartSLOduration=1.924411586 podStartE2EDuration="1.924411586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:48:27.922842139 +0000 UTC m=+784.861128935" watchObservedRunningTime="2026-11-02 09:48:27.924411586 +0000 UTC m=+784.862698362"
Nov 02 09:48:27.925440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:27.925421    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"
Nov 02 09:48:27.932385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:27.932361    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:48:27.932538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:27.932468    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:48:27.932590 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:27.932540    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:48:27.932590 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:27.932541    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:48:27.935833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:27.935811    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"]
Nov 02 09:48:28.000368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.000327    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6e3572f-rrqpx\" (UID: \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"
Nov 02 09:48:28.000549 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.000525    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6e3572f-rrqpx\" (UID: \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"
Nov 02 09:48:28.000758 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.000707    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpfbj\" (UniqueName: \"kubernetes.io/projected/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-kube-api-access-dpfbj\") pod \"maas-api-revoke-sub-c6e3572f-rrqpx\" (UID: \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"
Nov 02 09:48:28.102620 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.102580    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6e3572f-rrqpx\" (UID: \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"
Nov 02 09:48:28.102803 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.102686    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6e3572f-rrqpx\" (UID: \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"
Nov 02 09:48:28.102803 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.102799    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpfbj\" (UniqueName: \"kubernetes.io/projected/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-kube-api-access-dpfbj\") pod \"maas-api-revoke-sub-c6e3572f-rrqpx\" (UID: \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"
Nov 02 09:48:28.103529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.103507    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6e3572f-rrqpx\" (UID: \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"
Nov 02 09:48:28.105403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.105378    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6e3572f-rrqpx\" (UID: \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"
Nov 02 09:48:28.115666 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.115636    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpfbj\" (UniqueName: \"kubernetes.io/projected/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-kube-api-access-dpfbj\") pod \"maas-api-revoke-sub-c6e3572f-rrqpx\" (UID: \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"
Nov 02 09:48:28.241427 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.241346    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"
Nov 02 09:48:28.548814 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:48:28.548785    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac93e91_bcc2_4514_8f7b_15bbf0ca5286.slice/crio-a9c4c2af357d212b2a1daface7ee4dace8e56ed808dfbb2a508f01164810c724 WatchSource:0}: Error finding container a9c4c2af357d212b2a1daface7ee4dace8e56ed808dfbb2a508f01164810c724: Status 404 returned error can't find the container with id a9c4c2af357d212b2a1daface7ee4dace8e56ed808dfbb2a508f01164810c724
Nov 02 09:48:28.575517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.571852    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"]
Nov 02 09:48:28.703634 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.703604    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:28.705536 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.705509    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 02 09:48:28.705660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.705559    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm" podUID="f5071cf6-026d-407b-84b3-f682c15a8d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 02 09:48:28.909683 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:28.909600    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx" event={"ID":"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286","Type":"ContainerStarted","Data":"a9c4c2af357d212b2a1daface7ee4dace8e56ed808dfbb2a508f01164810c724"}
Nov 02 09:48:29.704734 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:29.704698    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 02 09:48:29.704861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:29.704754    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm" podUID="f5071cf6-026d-407b-84b3-f682c15a8d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 02 09:48:29.918645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:29.918557    4449 generic.go:356] "Generic (PLEG): container finished" podID="9ac93e91-bcc2-4514-8f7b-15bbf0ca5286" containerID="e6817f9b7dbc3f7445f9367ca84f7ea4f716b08491dc010d862eb7540f5b7bbf" exitCode=0
Nov 02 09:48:29.919021 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:29.918642    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx" event={"ID":"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286","Type":"ContainerDied","Data":"e6817f9b7dbc3f7445f9367ca84f7ea4f716b08491dc010d862eb7540f5b7bbf"}
Nov 02 09:48:30.704576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:30.704543    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 02 09:48:30.704748 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:30.704602    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm" podUID="f5071cf6-026d-407b-84b3-f682c15a8d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 02 09:48:31.122846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.122817    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"
Nov 02 09:48:31.283519 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.283430    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-openshift-ca\") pod \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\" (UID: \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\") "
Nov 02 09:48:31.283667 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.283533    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-kube-api-access-dpfbj\" (UniqueName: \"kubernetes.io/projected/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-kube-api-access-dpfbj\") pod \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\" (UID: \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\") "
Nov 02 09:48:31.283760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.283740    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-xks-ca\") pod \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\" (UID: \"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286\") "
Nov 02 09:48:31.283856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.283830    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-openshift-ca" pod "9ac93e91-bcc2-4514-8f7b-15bbf0ca5286" (UID: "9ac93e91-bcc2-4514-8f7b-15bbf0ca5286"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:48:31.284249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.284209    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:31.285799 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.285777    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-kube-api-access-dpfbj" pod "9ac93e91-bcc2-4514-8f7b-15bbf0ca5286" (UID: "9ac93e91-bcc2-4514-8f7b-15bbf0ca5286"). InnerVolumeSpecName "kube-api-access-dpfbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:48:31.285903 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.285818    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-xks-ca" pod "9ac93e91-bcc2-4514-8f7b-15bbf0ca5286" (UID: "9ac93e91-bcc2-4514-8f7b-15bbf0ca5286"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:48:31.385212 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.385173    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpfbj\" (UniqueName: \"kubernetes.io/projected/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-kube-api-access-dpfbj\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:31.385212 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.385209    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:31.704202 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.704166    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 02 09:48:31.704407 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.704275    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm" podUID="f5071cf6-026d-407b-84b3-f682c15a8d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 02 09:48:31.934772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.934746    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"
Nov 02 09:48:31.934910 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.934744    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx" event={"ID":"9ac93e91-bcc2-4514-8f7b-15bbf0ca5286","Type":"ContainerDied","Data":"a9c4c2af357d212b2a1daface7ee4dace8e56ed808dfbb2a508f01164810c724"}
Nov 02 09:48:31.934910 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:31.934865    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9c4c2af357d212b2a1daface7ee4dace8e56ed808dfbb2a508f01164810c724"
Nov 02 09:48:32.704372 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:32.704336    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 02 09:48:32.704851 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:32.704398    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm" podUID="f5071cf6-026d-407b-84b3-f682c15a8d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 02 09:48:33.709863 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:33.709831    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:33.710361 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:33.710256    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:33.710954 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:33.710931    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:48:35.967259 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:35.966424    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:48:36.048244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:36.047552    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 09:48:36.340243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:36.338805    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" podUID="e5827546-705c-4b91-a21e-daacfee4134a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Nov 02 09:48:36.572243 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:48:36.571483    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:48:38.212334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.212293    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"]
Nov 02 09:48:38.213050 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.213032    4449 cpu_manager.go:414] "containerMap: removing container" podUID="9ac93e91-bcc2-4514-8f7b-15bbf0ca5286" containerName="revoke-keys"
Nov 02 09:48:38.213097 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.213053    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac93e91-bcc2-4514-8f7b-15bbf0ca5286" containerName="revoke-keys"
Nov 02 09:48:38.213200 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.213190    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac93e91-bcc2-4514-8f7b-15bbf0ca5286" containerName="revoke-keys"
Nov 02 09:48:38.216782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.216764    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"
Nov 02 09:48:38.226823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.226795    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:48:38.227050 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.227026    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:48:38.227141 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.226909    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:48:38.227141 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.226858    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:48:38.229983 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.229946    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"]
Nov 02 09:48:38.303584 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.303543    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwlxw\" (UniqueName: \"kubernetes.io/projected/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-kube-api-access-dwlxw\") pod \"maas-api-revoke-sub-553494eb-xlm2n\" (UID: \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"
Nov 02 09:48:38.303584 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.303587    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-553494eb-xlm2n\" (UID: \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"
Nov 02 09:48:38.303790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.303722    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-553494eb-xlm2n\" (UID: \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"
Nov 02 09:48:38.404663 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.404623    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwlxw\" (UniqueName: \"kubernetes.io/projected/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-kube-api-access-dwlxw\") pod \"maas-api-revoke-sub-553494eb-xlm2n\" (UID: \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"
Nov 02 09:48:38.404876 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.404686    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-553494eb-xlm2n\" (UID: \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"
Nov 02 09:48:38.404876 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.404771    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-553494eb-xlm2n\" (UID: \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"
Nov 02 09:48:38.405402 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.405380    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-553494eb-xlm2n\" (UID: \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"
Nov 02 09:48:38.407349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.407324    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-553494eb-xlm2n\" (UID: \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"
Nov 02 09:48:38.417779 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.417732    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwlxw\" (UniqueName: \"kubernetes.io/projected/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-kube-api-access-dwlxw\") pod \"maas-api-revoke-sub-553494eb-xlm2n\" (UID: \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"
Nov 02 09:48:38.539209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.539168    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"
Nov 02 09:48:38.836918 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:38.836878    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"]
Nov 02 09:48:38.854330 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:48:38.854288    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01f7d9d_391b_4f4a_9cd0_31bff22bfd88.slice/crio-a452e9845cd69ac30b6e8b53f6a8f5b6720b361c5142e666b2a798fbb8d5247d WatchSource:0}: Error finding container a452e9845cd69ac30b6e8b53f6a8f5b6720b361c5142e666b2a798fbb8d5247d: Status 404 returned error can't find the container with id a452e9845cd69ac30b6e8b53f6a8f5b6720b361c5142e666b2a798fbb8d5247d
Nov 02 09:48:39.081207 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:39.081173    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n" event={"ID":"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88","Type":"ContainerStarted","Data":"a452e9845cd69ac30b6e8b53f6a8f5b6720b361c5142e666b2a798fbb8d5247d"}
Nov 02 09:48:40.091290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:40.091209    4449 generic.go:356] "Generic (PLEG): container finished" podID="e01f7d9d-391b-4f4a-9cd0-31bff22bfd88" containerID="e07064db4c1e99227ac245261b09566f26480d8f84f5ac7d8447782e4d1c5b38" exitCode=0
Nov 02 09:48:40.091771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:40.091340    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n" event={"ID":"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88","Type":"ContainerDied","Data":"e07064db4c1e99227ac245261b09566f26480d8f84f5ac7d8447782e4d1c5b38"}
Nov 02 09:48:41.288638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:41.288613    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"
Nov 02 09:48:41.346828 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:41.346774    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-xks-ca\") pod \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\" (UID: \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\") "
Nov 02 09:48:41.347032 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:41.346867    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-kube-api-access-dwlxw\" (UniqueName: \"kubernetes.io/projected/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-kube-api-access-dwlxw\") pod \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\" (UID: \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\") "
Nov 02 09:48:41.347032 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:41.346993    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-openshift-ca\") pod \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\" (UID: \"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88\") "
Nov 02 09:48:41.347464 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:41.347431    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-openshift-ca" pod "e01f7d9d-391b-4f4a-9cd0-31bff22bfd88" (UID: "e01f7d9d-391b-4f4a-9cd0-31bff22bfd88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:48:41.347823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:41.347801    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:41.349949 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:41.349862    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-kube-api-access-dwlxw" pod "e01f7d9d-391b-4f4a-9cd0-31bff22bfd88" (UID: "e01f7d9d-391b-4f4a-9cd0-31bff22bfd88"). InnerVolumeSpecName "kube-api-access-dwlxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:48:41.351709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:41.351662    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-xks-ca" pod "e01f7d9d-391b-4f4a-9cd0-31bff22bfd88" (UID: "e01f7d9d-391b-4f4a-9cd0-31bff22bfd88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:48:41.448921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:41.448885    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:41.448921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:41.448920    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwlxw\" (UniqueName: \"kubernetes.io/projected/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88-kube-api-access-dwlxw\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:42.108230 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:42.108197    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"
Nov 02 09:48:42.108455 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:42.108275    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n" event={"ID":"e01f7d9d-391b-4f4a-9cd0-31bff22bfd88","Type":"ContainerDied","Data":"a452e9845cd69ac30b6e8b53f6a8f5b6720b361c5142e666b2a798fbb8d5247d"}
Nov 02 09:48:42.108455 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:42.108362    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a452e9845cd69ac30b6e8b53f6a8f5b6720b361c5142e666b2a798fbb8d5247d"
Nov 02 09:48:51.167254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.167186    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"]
Nov 02 09:48:51.168090 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.168065    4449 cpu_manager.go:414] "containerMap: removing container" podUID="e01f7d9d-391b-4f4a-9cd0-31bff22bfd88" containerName="revoke-keys"
Nov 02 09:48:51.168201 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.168091    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e01f7d9d-391b-4f4a-9cd0-31bff22bfd88" containerName="revoke-keys"
Nov 02 09:48:51.168312 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.168292    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="e01f7d9d-391b-4f4a-9cd0-31bff22bfd88" containerName="revoke-keys"
Nov 02 09:48:51.173051 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.173029    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"
Nov 02 09:48:51.181682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.181651    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:48:51.181891 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.181672    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:48:51.181891 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.181676    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:48:51.182058 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.181728    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:48:51.182843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.182823    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"]
Nov 02 09:48:51.332907 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.332874    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17f4b7b2-2gvvc\" (UID: \"b1f895e8-16a7-4246-9722-c6955d275a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"
Nov 02 09:48:51.333408 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.333375    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17f4b7b2-2gvvc\" (UID: \"b1f895e8-16a7-4246-9722-c6955d275a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"
Nov 02 09:48:51.333551 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.333463    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zw4n\" (UniqueName: \"kubernetes.io/projected/b1f895e8-16a7-4246-9722-c6955d275a44-kube-api-access-4zw4n\") pod \"maas-api-revoke-sub-17f4b7b2-2gvvc\" (UID: \"b1f895e8-16a7-4246-9722-c6955d275a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"
Nov 02 09:48:51.434862 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.434764    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zw4n\" (UniqueName: \"kubernetes.io/projected/b1f895e8-16a7-4246-9722-c6955d275a44-kube-api-access-4zw4n\") pod \"maas-api-revoke-sub-17f4b7b2-2gvvc\" (UID: \"b1f895e8-16a7-4246-9722-c6955d275a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"
Nov 02 09:48:51.435043 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.434919    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17f4b7b2-2gvvc\" (UID: \"b1f895e8-16a7-4246-9722-c6955d275a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"
Nov 02 09:48:51.435122 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.435103    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17f4b7b2-2gvvc\" (UID: \"b1f895e8-16a7-4246-9722-c6955d275a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"
Nov 02 09:48:51.435683 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.435657    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17f4b7b2-2gvvc\" (UID: \"b1f895e8-16a7-4246-9722-c6955d275a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"
Nov 02 09:48:51.437654 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.437630    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17f4b7b2-2gvvc\" (UID: \"b1f895e8-16a7-4246-9722-c6955d275a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"
Nov 02 09:48:51.448137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.448107    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zw4n\" (UniqueName: \"kubernetes.io/projected/b1f895e8-16a7-4246-9722-c6955d275a44-kube-api-access-4zw4n\") pod \"maas-api-revoke-sub-17f4b7b2-2gvvc\" (UID: \"b1f895e8-16a7-4246-9722-c6955d275a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"
Nov 02 09:48:51.494420 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.494383    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"
Nov 02 09:48:51.808131 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:48:51.808075    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f895e8_16a7_4246_9722_c6955d275a44.slice/crio-45f91356c6dae1dd3e75c7ccacad1803d55ea87c77994f4a9e8d255750ac1db5 WatchSource:0}: Error finding container 45f91356c6dae1dd3e75c7ccacad1803d55ea87c77994f4a9e8d255750ac1db5: Status 404 returned error can't find the container with id 45f91356c6dae1dd3e75c7ccacad1803d55ea87c77994f4a9e8d255750ac1db5
Nov 02 09:48:51.819573 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:51.819151    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"]
Nov 02 09:48:52.203694 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:52.203650    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc" event={"ID":"b1f895e8-16a7-4246-9722-c6955d275a44","Type":"ContainerStarted","Data":"ece06224952f564bfc4446fd64441e0c835954693e865fdaab75feb1677adfbb"}
Nov 02 09:48:52.204149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:52.203701    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc" event={"ID":"b1f895e8-16a7-4246-9722-c6955d275a44","Type":"ContainerStarted","Data":"45f91356c6dae1dd3e75c7ccacad1803d55ea87c77994f4a9e8d255750ac1db5"}
Nov 02 09:48:52.233814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:52.233758    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc" podStartSLOduration=1.23374397 podStartE2EDuration="1.23374397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:48:52.230092316 +0000 UTC m=+809.168379102" watchObservedRunningTime="2026-11-02 09:48:52.23374397 +0000 UTC m=+809.172030746"
Nov 02 09:48:53.219733 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:53.219693    4449 generic.go:356] "Generic (PLEG): container finished" podID="b1f895e8-16a7-4246-9722-c6955d275a44" containerID="ece06224952f564bfc4446fd64441e0c835954693e865fdaab75feb1677adfbb" exitCode=0
Nov 02 09:48:53.220260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:53.219787    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc" event={"ID":"b1f895e8-16a7-4246-9722-c6955d275a44","Type":"ContainerDied","Data":"ece06224952f564bfc4446fd64441e0c835954693e865fdaab75feb1677adfbb"}
Nov 02 09:48:54.400206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:54.400175    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"
Nov 02 09:48:54.576306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:54.576267    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-openshift-ca\") pod \"b1f895e8-16a7-4246-9722-c6955d275a44\" (UID: \"b1f895e8-16a7-4246-9722-c6955d275a44\") "
Nov 02 09:48:54.576533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:54.576406    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f895e8-16a7-4246-9722-c6955d275a44-kube-api-access-4zw4n\" (UniqueName: \"kubernetes.io/projected/b1f895e8-16a7-4246-9722-c6955d275a44-kube-api-access-4zw4n\") pod \"b1f895e8-16a7-4246-9722-c6955d275a44\" (UID: \"b1f895e8-16a7-4246-9722-c6955d275a44\") "
Nov 02 09:48:54.576533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:54.576453    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-xks-ca\") pod \"b1f895e8-16a7-4246-9722-c6955d275a44\" (UID: \"b1f895e8-16a7-4246-9722-c6955d275a44\") "
Nov 02 09:48:54.576747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:54.576718    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-openshift-ca" pod "b1f895e8-16a7-4246-9722-c6955d275a44" (UID: "b1f895e8-16a7-4246-9722-c6955d275a44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:48:54.577135 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:54.577105    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:54.578778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:54.578756    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f895e8-16a7-4246-9722-c6955d275a44-kube-api-access-4zw4n" pod "b1f895e8-16a7-4246-9722-c6955d275a44" (UID: "b1f895e8-16a7-4246-9722-c6955d275a44"). InnerVolumeSpecName "kube-api-access-4zw4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:48:54.578853 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:54.578780    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-xks-ca" pod "b1f895e8-16a7-4246-9722-c6955d275a44" (UID: "b1f895e8-16a7-4246-9722-c6955d275a44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:48:54.678372 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:54.678329    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zw4n\" (UniqueName: \"kubernetes.io/projected/b1f895e8-16a7-4246-9722-c6955d275a44-kube-api-access-4zw4n\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:54.678372 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:54.678364    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f895e8-16a7-4246-9722-c6955d275a44-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:48:55.244550 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:55.244517    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"
Nov 02 09:48:55.244721 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:55.244557    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc" event={"ID":"b1f895e8-16a7-4246-9722-c6955d275a44","Type":"ContainerDied","Data":"45f91356c6dae1dd3e75c7ccacad1803d55ea87c77994f4a9e8d255750ac1db5"}
Nov 02 09:48:55.244721 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:48:55.244635    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45f91356c6dae1dd3e75c7ccacad1803d55ea87c77994f4a9e8d255750ac1db5"
Nov 02 09:49:00.064569 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.064521    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"]
Nov 02 09:49:00.412821 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.412741    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"]
Nov 02 09:49:00.413469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.413455    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b1f895e8-16a7-4246-9722-c6955d275a44" containerName="revoke-keys"
Nov 02 09:49:00.413522 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.413471    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f895e8-16a7-4246-9722-c6955d275a44" containerName="revoke-keys"
Nov 02 09:49:00.413597 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.413587    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f895e8-16a7-4246-9722-c6955d275a44" containerName="revoke-keys"
Nov 02 09:49:00.418577 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.418554    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"
Nov 02 09:49:00.425925 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.425867    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:49:00.426132 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.425977    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:49:00.426238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.426161    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:49:00.426332 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.426251    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:49:00.433631 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.433599    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"]
Nov 02 09:49:00.527249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.527179    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkkfl\" (UniqueName: \"kubernetes.io/projected/695c1281-51d3-45c8-85f2-04838c8bce77-kube-api-access-qkkfl\") pod \"maas-api-revoke-sub-2c9a8d23-tkkq9\" (UID: \"695c1281-51d3-45c8-85f2-04838c8bce77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"
Nov 02 09:49:00.527428 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.527344    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9a8d23-tkkq9\" (UID: \"695c1281-51d3-45c8-85f2-04838c8bce77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"
Nov 02 09:49:00.527478 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.527460    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9a8d23-tkkq9\" (UID: \"695c1281-51d3-45c8-85f2-04838c8bce77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"
Nov 02 09:49:00.629480 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.629438    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9a8d23-tkkq9\" (UID: \"695c1281-51d3-45c8-85f2-04838c8bce77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"
Nov 02 09:49:00.629663 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.629605    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkkfl\" (UniqueName: \"kubernetes.io/projected/695c1281-51d3-45c8-85f2-04838c8bce77-kube-api-access-qkkfl\") pod \"maas-api-revoke-sub-2c9a8d23-tkkq9\" (UID: \"695c1281-51d3-45c8-85f2-04838c8bce77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"
Nov 02 09:49:00.629663 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.629641    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9a8d23-tkkq9\" (UID: \"695c1281-51d3-45c8-85f2-04838c8bce77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"
Nov 02 09:49:00.630142 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.630115    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9a8d23-tkkq9\" (UID: \"695c1281-51d3-45c8-85f2-04838c8bce77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"
Nov 02 09:49:00.632227 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.632190    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9a8d23-tkkq9\" (UID: \"695c1281-51d3-45c8-85f2-04838c8bce77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"
Nov 02 09:49:00.645985 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.645957    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkkfl\" (UniqueName: \"kubernetes.io/projected/695c1281-51d3-45c8-85f2-04838c8bce77-kube-api-access-qkkfl\") pod \"maas-api-revoke-sub-2c9a8d23-tkkq9\" (UID: \"695c1281-51d3-45c8-85f2-04838c8bce77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"
Nov 02 09:49:00.735362 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:00.735272    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"
Nov 02 09:49:01.024042 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:49:01.024001    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695c1281_51d3_45c8_85f2_04838c8bce77.slice/crio-4b418dd7c36ec7194cf4132525224005c8c2cbc6f3670047e54f099c8e7adb3c WatchSource:0}: Error finding container 4b418dd7c36ec7194cf4132525224005c8c2cbc6f3670047e54f099c8e7adb3c: Status 404 returned error can't find the container with id 4b418dd7c36ec7194cf4132525224005c8c2cbc6f3670047e54f099c8e7adb3c
Nov 02 09:49:01.024641 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:01.024617    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"]
Nov 02 09:49:01.317017 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:01.316979    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9" event={"ID":"695c1281-51d3-45c8-85f2-04838c8bce77","Type":"ContainerStarted","Data":"4b418dd7c36ec7194cf4132525224005c8c2cbc6f3670047e54f099c8e7adb3c"}
Nov 02 09:49:02.327745 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:02.327702    4449 generic.go:356] "Generic (PLEG): container finished" podID="695c1281-51d3-45c8-85f2-04838c8bce77" containerID="575cac6caa5982a6c4fb4239c68c9b065a078f41396381c311db9391663a088a" exitCode=0
Nov 02 09:49:02.328150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:02.327791    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9" event={"ID":"695c1281-51d3-45c8-85f2-04838c8bce77","Type":"ContainerDied","Data":"575cac6caa5982a6c4fb4239c68c9b065a078f41396381c311db9391663a088a"}
Nov 02 09:49:03.515755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:03.515725    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"
Nov 02 09:49:03.696814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:03.696718    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695c1281-51d3-45c8-85f2-04838c8bce77-kube-api-access-qkkfl\" (UniqueName: \"kubernetes.io/projected/695c1281-51d3-45c8-85f2-04838c8bce77-kube-api-access-qkkfl\") pod \"695c1281-51d3-45c8-85f2-04838c8bce77\" (UID: \"695c1281-51d3-45c8-85f2-04838c8bce77\") "
Nov 02 09:49:03.696814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:03.696764    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-xks-ca\") pod \"695c1281-51d3-45c8-85f2-04838c8bce77\" (UID: \"695c1281-51d3-45c8-85f2-04838c8bce77\") "
Nov 02 09:49:03.696814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:03.696784    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-openshift-ca\") pod \"695c1281-51d3-45c8-85f2-04838c8bce77\" (UID: \"695c1281-51d3-45c8-85f2-04838c8bce77\") "
Nov 02 09:49:03.697277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:03.697206    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-openshift-ca" pod "695c1281-51d3-45c8-85f2-04838c8bce77" (UID: "695c1281-51d3-45c8-85f2-04838c8bce77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:03.697506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:03.697481    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:03.699497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:03.699466    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695c1281-51d3-45c8-85f2-04838c8bce77-kube-api-access-qkkfl" pod "695c1281-51d3-45c8-85f2-04838c8bce77" (UID: "695c1281-51d3-45c8-85f2-04838c8bce77"). InnerVolumeSpecName "kube-api-access-qkkfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:03.699497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:03.699476    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-xks-ca" pod "695c1281-51d3-45c8-85f2-04838c8bce77" (UID: "695c1281-51d3-45c8-85f2-04838c8bce77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:49:03.798452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:03.798414    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkkfl\" (UniqueName: \"kubernetes.io/projected/695c1281-51d3-45c8-85f2-04838c8bce77-kube-api-access-qkkfl\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:03.798452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:03.798445    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695c1281-51d3-45c8-85f2-04838c8bce77-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:04.348982 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:04.348940    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"
Nov 02 09:49:04.349174 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:04.348971    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9" event={"ID":"695c1281-51d3-45c8-85f2-04838c8bce77","Type":"ContainerDied","Data":"4b418dd7c36ec7194cf4132525224005c8c2cbc6f3670047e54f099c8e7adb3c"}
Nov 02 09:49:04.349174 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:04.349036    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b418dd7c36ec7194cf4132525224005c8c2cbc6f3670047e54f099c8e7adb3c"
Nov 02 09:49:05.064915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.064859    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp" podUID="25b927fd-89fb-40ba-bf6c-b4ca5d926ab7" containerName="payload-processing" containerID="cri-o://f31ca3a1439e6f63c1d95ad7e68dbc8e6b1d236059284465cead3c416b83976b" gracePeriod=25
Nov 02 09:49:05.338468 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.338438    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:49:05.358436 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.358400    4449 generic.go:356] "Generic (PLEG): container finished" podID="25b927fd-89fb-40ba-bf6c-b4ca5d926ab7" containerID="f31ca3a1439e6f63c1d95ad7e68dbc8e6b1d236059284465cead3c416b83976b" exitCode=0
Nov 02 09:49:05.358636 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.358481    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"
Nov 02 09:49:05.358636 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.358483    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp" event={"ID":"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7","Type":"ContainerDied","Data":"f31ca3a1439e6f63c1d95ad7e68dbc8e6b1d236059284465cead3c416b83976b"}
Nov 02 09:49:05.358636 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.358617    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp" event={"ID":"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7","Type":"ContainerDied","Data":"3519ae36dfe3bf60107d9d994dae2ba78a99f246f2dd8e694c2e4d7d51784b26"}
Nov 02 09:49:05.358790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.358647    4449 scope.go:122] "RemoveContainer" containerID="f31ca3a1439e6f63c1d95ad7e68dbc8e6b1d236059284465cead3c416b83976b"
Nov 02 09:49:05.387339 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.387314    4449 scope.go:122] "RemoveContainer" containerID="f31ca3a1439e6f63c1d95ad7e68dbc8e6b1d236059284465cead3c416b83976b"
Nov 02 09:49:05.387705 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:49:05.387671    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f31ca3a1439e6f63c1d95ad7e68dbc8e6b1d236059284465cead3c416b83976b\": container with ID starting with f31ca3a1439e6f63c1d95ad7e68dbc8e6b1d236059284465cead3c416b83976b not found: ID does not exist" containerID="f31ca3a1439e6f63c1d95ad7e68dbc8e6b1d236059284465cead3c416b83976b"
Nov 02 09:49:05.387815 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.387705    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31ca3a1439e6f63c1d95ad7e68dbc8e6b1d236059284465cead3c416b83976b"} err="failed to get container status \"f31ca3a1439e6f63c1d95ad7e68dbc8e6b1d236059284465cead3c416b83976b\": rpc error: code = NotFound desc = could not find container \"f31ca3a1439e6f63c1d95ad7e68dbc8e6b1d236059284465cead3c416b83976b\": container with ID starting with f31ca3a1439e6f63c1d95ad7e68dbc8e6b1d236059284465cead3c416b83976b not found: ID does not exist"
Nov 02 09:49:05.417460 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.417419    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-routing-overlay-volume\") pod \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") "
Nov 02 09:49:05.417631 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.417470    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-kube-api-access-6dmrz\" (UniqueName: \"kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-kube-api-access-6dmrz\") pod \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") "
Nov 02 09:49:05.417672 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.417660    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-plugins-config-volume\") pod \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") "
Nov 02 09:49:05.417710 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.417697    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-provider-credentials\") pod \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\" (UID: \"25b927fd-89fb-40ba-bf6c-b4ca5d926ab7\") "
Nov 02 09:49:05.417985 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.417886    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-routing-overlay-volume" pod "25b927fd-89fb-40ba-bf6c-b4ca5d926ab7" (UID: "25b927fd-89fb-40ba-bf6c-b4ca5d926ab7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:05.418129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.418079    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-plugins-config-volume" pod "25b927fd-89fb-40ba-bf6c-b4ca5d926ab7" (UID: "25b927fd-89fb-40ba-bf6c-b4ca5d926ab7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:05.418176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.418141    4449 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-routing-overlay-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:05.420094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.420064    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-kube-api-access-6dmrz" pod "25b927fd-89fb-40ba-bf6c-b4ca5d926ab7" (UID: "25b927fd-89fb-40ba-bf6c-b4ca5d926ab7"). InnerVolumeSpecName "kube-api-access-6dmrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:05.420236 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.420106    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-provider-credentials" pod "25b927fd-89fb-40ba-bf6c-b4ca5d926ab7" (UID: "25b927fd-89fb-40ba-bf6c-b4ca5d926ab7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:05.519014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.518969    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:05.519014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.519003    4449 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-provider-credentials\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:05.519014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.519019    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dmrz\" (UniqueName: \"kubernetes.io/projected/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7-kube-api-access-6dmrz\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:05.689065 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.689026    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"]
Nov 02 09:49:05.695703 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:05.695663    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85982dgjp"]
Nov 02 09:49:06.592147 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:49:06.592099    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:49:07.230700 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:07.230661    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b927fd-89fb-40ba-bf6c-b4ca5d926ab7" path="/var/lib/kubelet/pods/25b927fd-89fb-40ba-bf6c-b4ca5d926ab7/volumes"
Nov 02 09:49:11.752134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.752090    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"]
Nov 02 09:49:11.752798 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.752778    4449 cpu_manager.go:414] "containerMap: removing container" podUID="25b927fd-89fb-40ba-bf6c-b4ca5d926ab7" containerName="payload-processing"
Nov 02 09:49:11.752798 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.752795    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b927fd-89fb-40ba-bf6c-b4ca5d926ab7" containerName="payload-processing"
Nov 02 09:49:11.752885 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.752846    4449 cpu_manager.go:414] "containerMap: removing container" podUID="695c1281-51d3-45c8-85f2-04838c8bce77" containerName="revoke-keys"
Nov 02 09:49:11.752885 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.752853    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695c1281-51d3-45c8-85f2-04838c8bce77" containerName="revoke-keys"
Nov 02 09:49:11.752959 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.752948    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="695c1281-51d3-45c8-85f2-04838c8bce77" containerName="revoke-keys"
Nov 02 09:49:11.752991 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.752966    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b927fd-89fb-40ba-bf6c-b4ca5d926ab7" containerName="payload-processing"
Nov 02 09:49:11.756909 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.756891    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:11.763846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.763823    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-1521404c1537\""
Nov 02 09:49:11.768542 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.768514    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-4a99520c97db\""
Nov 02 09:49:11.774789 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.774746    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"]
Nov 02 09:49:11.778015 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.777968    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-plugins-e2e-extmod-b-204fba\""
Nov 02 09:49:11.778258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.778113    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"routing-overlay\""
Nov 02 09:49:11.778258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.778201    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"kube-root-ca.crt\""
Nov 02 09:49:11.778411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.778279    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"openshift-service-ca.crt\""
Nov 02 09:49:11.849099 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.849062    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:11.849432 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.849127    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:11.849432 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.849268    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxmn7\" (UniqueName: \"kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-kube-api-access-xxmn7\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:11.849432 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.849307    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:11.863947 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.863919    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-e2e--e7669bbb-dockercfg-zngpv\""
Nov 02 09:49:11.864130 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.864045    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-11e3267f4507\""
Nov 02 09:49:11.950150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.950115    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:11.950349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.950236    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:11.950349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.950283    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:11.950464 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.950420    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxmn7\" (UniqueName: \"kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-kube-api-access-xxmn7\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:11.951018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.950988    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:11.951150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.950995    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:11.953024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.953003    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:11.968548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:11.968518    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxmn7\" (UniqueName: \"kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-kube-api-access-xxmn7\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:12.073619 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.073577    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:12.371260 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:49:12.371202    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c609200_d239_4c5a_8743_acc3d200579f.slice/crio-86c91d5ada0ca29df4905e2dce1e50d558afaab0579d54c400e2ffab6b6a7d84 WatchSource:0}: Error finding container 86c91d5ada0ca29df4905e2dce1e50d558afaab0579d54c400e2ffab6b6a7d84: Status 404 returned error can't find the container with id 86c91d5ada0ca29df4905e2dce1e50d558afaab0579d54c400e2ffab6b6a7d84
Nov 02 09:49:12.371418 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.371400    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"]
Nov 02 09:49:12.549782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.549747    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq" event={"ID":"0c609200-d239-4c5a-8743-acc3d200579f","Type":"ContainerStarted","Data":"bad54496c3f320c98c1c3c463c93e0f17fae7fca5539d74ba08150f7dcbb3034"}
Nov 02 09:49:12.549960 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.549806    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq" event={"ID":"0c609200-d239-4c5a-8743-acc3d200579f","Type":"ContainerStarted","Data":"86c91d5ada0ca29df4905e2dce1e50d558afaab0579d54c400e2ffab6b6a7d84"}
Nov 02 09:49:12.549960 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.549826    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:12.576253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.576184    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq" podStartSLOduration=1.576169985 podStartE2EDuration="1.576169985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:12.573869676 +0000 UTC m=+829.512156451" watchObservedRunningTime="2026-11-02 09:49:12.576169985 +0000 UTC m=+829.514456762"
Nov 02 09:49:12.666630 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.666543    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"]
Nov 02 09:49:12.671043 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.671023    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:12.681129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.679833    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-plugins-e2e-extmod-a-629129\""
Nov 02 09:49:12.681129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.680080    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"openshift-service-ca.crt\""
Nov 02 09:49:12.681129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.680191    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"routing-overlay\""
Nov 02 09:49:12.681129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.680319    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"kube-root-ca.crt\""
Nov 02 09:49:12.681129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.680351    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-0c2b7c7653b8\""
Nov 02 09:49:12.681129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.680477    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-e2e--7be3f336-dockercfg-2rxpp\""
Nov 02 09:49:12.698247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.686805    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"]
Nov 02 09:49:12.759666 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.759634    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzzss\" (UniqueName: \"kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-kube-api-access-vzzss\") pod \"payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:12.760084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.759760    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:12.760084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.759855    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:12.760084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.759907    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:12.861129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.861089    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:12.861373 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.861136    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:12.861373 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.861342    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzzss\" (UniqueName: \"kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-kube-api-access-vzzss\") pod \"payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:12.861500 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.861463    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:12.861883 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.861862    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:12.861994 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.861977    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:12.864017 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.863999    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:12.874916 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:12.874886    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzzss\" (UniqueName: \"kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-kube-api-access-vzzss\") pod \"payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:13.000708 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.000532    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:13.026965 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.026934    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"]
Nov 02 09:49:13.038493 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.038463    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"]
Nov 02 09:49:13.038650 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.038574    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"
Nov 02 09:49:13.061251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.056924    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:49:13.061251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.057010    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:49:13.061251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.057102    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:49:13.061251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.057150    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:49:13.167338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.167301    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1726b246-qwpvj\" (UID: \"63601e9d-7e10-406e-b77d-c30f26e519c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"
Nov 02 09:49:13.167496 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.167405    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gqcr\" (UniqueName: \"kubernetes.io/projected/63601e9d-7e10-406e-b77d-c30f26e519c1-kube-api-access-4gqcr\") pod \"maas-api-revoke-sub-1726b246-qwpvj\" (UID: \"63601e9d-7e10-406e-b77d-c30f26e519c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"
Nov 02 09:49:13.167588 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.167556    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1726b246-qwpvj\" (UID: \"63601e9d-7e10-406e-b77d-c30f26e519c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"
Nov 02 09:49:13.268497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.268399    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1726b246-qwpvj\" (UID: \"63601e9d-7e10-406e-b77d-c30f26e519c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"
Nov 02 09:49:13.268679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.268515    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gqcr\" (UniqueName: \"kubernetes.io/projected/63601e9d-7e10-406e-b77d-c30f26e519c1-kube-api-access-4gqcr\") pod \"maas-api-revoke-sub-1726b246-qwpvj\" (UID: \"63601e9d-7e10-406e-b77d-c30f26e519c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"
Nov 02 09:49:13.268679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.268573    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1726b246-qwpvj\" (UID: \"63601e9d-7e10-406e-b77d-c30f26e519c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"
Nov 02 09:49:13.269377 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.269355    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1726b246-qwpvj\" (UID: \"63601e9d-7e10-406e-b77d-c30f26e519c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"
Nov 02 09:49:13.271074 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.271053    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1726b246-qwpvj\" (UID: \"63601e9d-7e10-406e-b77d-c30f26e519c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"
Nov 02 09:49:13.281430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.281396    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gqcr\" (UniqueName: \"kubernetes.io/projected/63601e9d-7e10-406e-b77d-c30f26e519c1-kube-api-access-4gqcr\") pod \"maas-api-revoke-sub-1726b246-qwpvj\" (UID: \"63601e9d-7e10-406e-b77d-c30f26e519c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"
Nov 02 09:49:13.341766 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:49:13.341737    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a59ba0_c7ea_4a09_81ea_166675303dc3.slice/crio-7ac10331b9524bbabcc68c87f3c6419b0155e13157cfdecf676a8d81cd51cf48 WatchSource:0}: Error finding container 7ac10331b9524bbabcc68c87f3c6419b0155e13157cfdecf676a8d81cd51cf48: Status 404 returned error can't find the container with id 7ac10331b9524bbabcc68c87f3c6419b0155e13157cfdecf676a8d81cd51cf48
Nov 02 09:49:13.342378 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.342350    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"]
Nov 02 09:49:13.414913 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.414878    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"
Nov 02 09:49:13.561350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.561313    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m" event={"ID":"26a59ba0-c7ea-4a09-81ea-166675303dc3","Type":"ContainerStarted","Data":"ec18c5269381c737c53437efb669909346686cab789eec82b2fa5990cea0b168"}
Nov 02 09:49:13.561535 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.561384    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:13.561535 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.561412    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m" event={"ID":"26a59ba0-c7ea-4a09-81ea-166675303dc3","Type":"ContainerStarted","Data":"7ac10331b9524bbabcc68c87f3c6419b0155e13157cfdecf676a8d81cd51cf48"}
Nov 02 09:49:13.584973 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.584919    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m" podStartSLOduration=1.584905033 podStartE2EDuration="1.584905033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:13.580972343 +0000 UTC m=+830.519259131" watchObservedRunningTime="2026-11-02 09:49:13.584905033 +0000 UTC m=+830.523191808"
Nov 02 09:49:13.721361 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:13.721322    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"]
Nov 02 09:49:14.572232 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:14.572173    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj" event={"ID":"63601e9d-7e10-406e-b77d-c30f26e519c1","Type":"ContainerStarted","Data":"65f49c7709a70ccad45dec5858d862ae336ee26feaf9c8e5a242968fa198c468"}
Nov 02 09:49:14.572649 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:14.572261    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj" event={"ID":"63601e9d-7e10-406e-b77d-c30f26e519c1","Type":"ContainerStarted","Data":"f0d8c6d06419402f50c70e9bdeb313b688420eb68c646c5e59fd92b5a5d3de9c"}
Nov 02 09:49:14.597092 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:14.596988    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj" podStartSLOduration=1.5969702080000001 podStartE2EDuration="1.596970208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:14.591518764 +0000 UTC m=+831.529805563" watchObservedRunningTime="2026-11-02 09:49:14.596970208 +0000 UTC m=+831.535257022"
Nov 02 09:49:15.582301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:15.582260    4449 generic.go:356] "Generic (PLEG): container finished" podID="63601e9d-7e10-406e-b77d-c30f26e519c1" containerID="65f49c7709a70ccad45dec5858d862ae336ee26feaf9c8e5a242968fa198c468" exitCode=0
Nov 02 09:49:15.582804 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:15.582316    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj" event={"ID":"63601e9d-7e10-406e-b77d-c30f26e519c1","Type":"ContainerDied","Data":"65f49c7709a70ccad45dec5858d862ae336ee26feaf9c8e5a242968fa198c468"}
Nov 02 09:49:16.748638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:16.748610    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"
Nov 02 09:49:16.828818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:16.828780    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-openshift-ca\") pod \"63601e9d-7e10-406e-b77d-c30f26e519c1\" (UID: \"63601e9d-7e10-406e-b77d-c30f26e519c1\") "
Nov 02 09:49:16.829046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:16.828908    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63601e9d-7e10-406e-b77d-c30f26e519c1-kube-api-access-4gqcr\" (UniqueName: \"kubernetes.io/projected/63601e9d-7e10-406e-b77d-c30f26e519c1-kube-api-access-4gqcr\") pod \"63601e9d-7e10-406e-b77d-c30f26e519c1\" (UID: \"63601e9d-7e10-406e-b77d-c30f26e519c1\") "
Nov 02 09:49:16.829046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:16.828971    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-xks-ca\") pod \"63601e9d-7e10-406e-b77d-c30f26e519c1\" (UID: \"63601e9d-7e10-406e-b77d-c30f26e519c1\") "
Nov 02 09:49:16.829165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:16.829131    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-openshift-ca" pod "63601e9d-7e10-406e-b77d-c30f26e519c1" (UID: "63601e9d-7e10-406e-b77d-c30f26e519c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:16.829467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:16.829433    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:16.831786 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:16.831747    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-xks-ca" pod "63601e9d-7e10-406e-b77d-c30f26e519c1" (UID: "63601e9d-7e10-406e-b77d-c30f26e519c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:49:16.832358 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:16.832333    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63601e9d-7e10-406e-b77d-c30f26e519c1-kube-api-access-4gqcr" pod "63601e9d-7e10-406e-b77d-c30f26e519c1" (UID: "63601e9d-7e10-406e-b77d-c30f26e519c1"). InnerVolumeSpecName "kube-api-access-4gqcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:16.930951 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:16.930905    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gqcr\" (UniqueName: \"kubernetes.io/projected/63601e9d-7e10-406e-b77d-c30f26e519c1-kube-api-access-4gqcr\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:16.930951 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:16.930948    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63601e9d-7e10-406e-b77d-c30f26e519c1-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:17.601108 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:17.601066    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj" event={"ID":"63601e9d-7e10-406e-b77d-c30f26e519c1","Type":"ContainerDied","Data":"f0d8c6d06419402f50c70e9bdeb313b688420eb68c646c5e59fd92b5a5d3de9c"}
Nov 02 09:49:17.601346 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:17.601117    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"
Nov 02 09:49:17.601346 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:17.601136    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0d8c6d06419402f50c70e9bdeb313b688420eb68c646c5e59fd92b5a5d3de9c"
Nov 02 09:49:18.104938 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:18.104897    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"]
Nov 02 09:49:18.108673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:18.108645    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:21.797424 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:21.797359    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:21.859746 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:21.859699    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz" podUID="5ed165dd-1500-4fd4-a5ae-5a35d894726e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:22.037841 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:22.037796    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:23.106326 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.106048    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m" podUID="26a59ba0-c7ea-4a09-81ea-166675303dc3" containerName="payload-processing" containerID="cri-o://ec18c5269381c737c53437efb669909346686cab789eec82b2fa5990cea0b168" gracePeriod=25
Nov 02 09:49:23.403318 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.403291    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:23.477792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.477753    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-routing-overlay-volume\") pod \"26a59ba0-c7ea-4a09-81ea-166675303dc3\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") "
Nov 02 09:49:23.477961 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.477800    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-kube-api-access-vzzss\" (UniqueName: \"kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-kube-api-access-vzzss\") pod \"26a59ba0-c7ea-4a09-81ea-166675303dc3\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") "
Nov 02 09:49:23.478062 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.478045    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-provider-credentials\") pod \"26a59ba0-c7ea-4a09-81ea-166675303dc3\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") "
Nov 02 09:49:23.478179 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.478153    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-routing-overlay-volume" pod "26a59ba0-c7ea-4a09-81ea-166675303dc3" (UID: "26a59ba0-c7ea-4a09-81ea-166675303dc3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:23.478322 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.478169    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-plugins-config-volume\") pod \"26a59ba0-c7ea-4a09-81ea-166675303dc3\" (UID: \"26a59ba0-c7ea-4a09-81ea-166675303dc3\") "
Nov 02 09:49:23.478535 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.478514    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-plugins-config-volume" pod "26a59ba0-c7ea-4a09-81ea-166675303dc3" (UID: "26a59ba0-c7ea-4a09-81ea-166675303dc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:23.479135 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.479111    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:23.479197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.479145    4449 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26a59ba0-c7ea-4a09-81ea-166675303dc3-routing-overlay-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:23.480603 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.480559    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-provider-credentials" pod "26a59ba0-c7ea-4a09-81ea-166675303dc3" (UID: "26a59ba0-c7ea-4a09-81ea-166675303dc3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:23.480603 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.480590    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-kube-api-access-vzzss" pod "26a59ba0-c7ea-4a09-81ea-166675303dc3" (UID: "26a59ba0-c7ea-4a09-81ea-166675303dc3"). InnerVolumeSpecName "kube-api-access-vzzss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:23.565585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.565555    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:49:23.580162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.580126    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzzss\" (UniqueName: \"kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-kube-api-access-vzzss\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:23.580162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.580159    4449 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26a59ba0-c7ea-4a09-81ea-166675303dc3-provider-credentials\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:23.790210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.790170    4449 generic.go:356] "Generic (PLEG): container finished" podID="26a59ba0-c7ea-4a09-81ea-166675303dc3" containerID="ec18c5269381c737c53437efb669909346686cab789eec82b2fa5990cea0b168" exitCode=0
Nov 02 09:49:23.790485 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.790288    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"
Nov 02 09:49:23.790785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.790309    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m" event={"ID":"26a59ba0-c7ea-4a09-81ea-166675303dc3","Type":"ContainerDied","Data":"ec18c5269381c737c53437efb669909346686cab789eec82b2fa5990cea0b168"}
Nov 02 09:49:23.790867 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.790838    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m" event={"ID":"26a59ba0-c7ea-4a09-81ea-166675303dc3","Type":"ContainerDied","Data":"7ac10331b9524bbabcc68c87f3c6419b0155e13157cfdecf676a8d81cd51cf48"}
Nov 02 09:49:23.790927 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.790875    4449 scope.go:122] "RemoveContainer" containerID="ec18c5269381c737c53437efb669909346686cab789eec82b2fa5990cea0b168"
Nov 02 09:49:23.811232 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.811190    4449 scope.go:122] "RemoveContainer" containerID="ec18c5269381c737c53437efb669909346686cab789eec82b2fa5990cea0b168"
Nov 02 09:49:23.811617 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:49:23.811587    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec18c5269381c737c53437efb669909346686cab789eec82b2fa5990cea0b168\": container with ID starting with ec18c5269381c737c53437efb669909346686cab789eec82b2fa5990cea0b168 not found: ID does not exist" containerID="ec18c5269381c737c53437efb669909346686cab789eec82b2fa5990cea0b168"
Nov 02 09:49:23.811707 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.811628    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec18c5269381c737c53437efb669909346686cab789eec82b2fa5990cea0b168"} err="failed to get container status \"ec18c5269381c737c53437efb669909346686cab789eec82b2fa5990cea0b168\": rpc error: code = NotFound desc = could not find container \"ec18c5269381c737c53437efb669909346686cab789eec82b2fa5990cea0b168\": container with ID starting with ec18c5269381c737c53437efb669909346686cab789eec82b2fa5990cea0b168 not found: ID does not exist"
Nov 02 09:49:23.823203 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.823162    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"]
Nov 02 09:49:23.827877 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:23.827853    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-86b9tgm4m"]
Nov 02 09:49:24.697125 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.697085    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"]
Nov 02 09:49:24.697779 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.697763    4449 cpu_manager.go:414] "containerMap: removing container" podUID="26a59ba0-c7ea-4a09-81ea-166675303dc3" containerName="payload-processing"
Nov 02 09:49:24.697842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.697781    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a59ba0-c7ea-4a09-81ea-166675303dc3" containerName="payload-processing"
Nov 02 09:49:24.697842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.697822    4449 cpu_manager.go:414] "containerMap: removing container" podUID="63601e9d-7e10-406e-b77d-c30f26e519c1" containerName="revoke-keys"
Nov 02 09:49:24.697842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.697827    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63601e9d-7e10-406e-b77d-c30f26e519c1" containerName="revoke-keys"
Nov 02 09:49:24.697934 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.697923    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a59ba0-c7ea-4a09-81ea-166675303dc3" containerName="payload-processing"
Nov 02 09:49:24.697968 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.697939    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="63601e9d-7e10-406e-b77d-c30f26e519c1" containerName="revoke-keys"
Nov 02 09:49:24.703457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.703432    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"
Nov 02 09:49:24.708563 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.708534    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:49:24.711076 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.711049    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:49:24.711418 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.711388    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:49:24.711969 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.711945    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:49:24.716553 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.715604    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"]
Nov 02 09:49:24.796667 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.796631    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda43351-8185-454d-898c-a5b118892ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e4d567b-rrm86\" (UID: \"fda43351-8185-454d-898c-a5b118892ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"
Nov 02 09:49:24.796839 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.796675    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsl9p\" (UniqueName: \"kubernetes.io/projected/fda43351-8185-454d-898c-a5b118892ac3-kube-api-access-gsl9p\") pod \"maas-api-revoke-sub-9e4d567b-rrm86\" (UID: \"fda43351-8185-454d-898c-a5b118892ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"
Nov 02 09:49:24.796928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.796849    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda43351-8185-454d-898c-a5b118892ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e4d567b-rrm86\" (UID: \"fda43351-8185-454d-898c-a5b118892ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"
Nov 02 09:49:24.898760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.898721    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda43351-8185-454d-898c-a5b118892ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e4d567b-rrm86\" (UID: \"fda43351-8185-454d-898c-a5b118892ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"
Nov 02 09:49:24.898760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.898761    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsl9p\" (UniqueName: \"kubernetes.io/projected/fda43351-8185-454d-898c-a5b118892ac3-kube-api-access-gsl9p\") pod \"maas-api-revoke-sub-9e4d567b-rrm86\" (UID: \"fda43351-8185-454d-898c-a5b118892ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"
Nov 02 09:49:24.898988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.898950    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda43351-8185-454d-898c-a5b118892ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e4d567b-rrm86\" (UID: \"fda43351-8185-454d-898c-a5b118892ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"
Nov 02 09:49:24.899620 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.899545    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda43351-8185-454d-898c-a5b118892ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e4d567b-rrm86\" (UID: \"fda43351-8185-454d-898c-a5b118892ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"
Nov 02 09:49:24.901329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.901309    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda43351-8185-454d-898c-a5b118892ac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e4d567b-rrm86\" (UID: \"fda43351-8185-454d-898c-a5b118892ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"
Nov 02 09:49:24.912249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:24.912204    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsl9p\" (UniqueName: \"kubernetes.io/projected/fda43351-8185-454d-898c-a5b118892ac3-kube-api-access-gsl9p\") pod \"maas-api-revoke-sub-9e4d567b-rrm86\" (UID: \"fda43351-8185-454d-898c-a5b118892ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"
Nov 02 09:49:25.025067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:25.024975    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"
Nov 02 09:49:25.208104 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:25.208072    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a59ba0-c7ea-4a09-81ea-166675303dc3" path="/var/lib/kubelet/pods/26a59ba0-c7ea-4a09-81ea-166675303dc3/volumes"
Nov 02 09:49:25.310744 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:49:25.310700    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda43351_8185_454d_898c_a5b118892ac3.slice/crio-51929b67b3f75c3126a93374c36d8b8aea50f904bea4ef494bd1712a43cbf559 WatchSource:0}: Error finding container 51929b67b3f75c3126a93374c36d8b8aea50f904bea4ef494bd1712a43cbf559: Status 404 returned error can't find the container with id 51929b67b3f75c3126a93374c36d8b8aea50f904bea4ef494bd1712a43cbf559
Nov 02 09:49:25.311529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:25.311501    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"]
Nov 02 09:49:25.813142 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:25.813107    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86" event={"ID":"fda43351-8185-454d-898c-a5b118892ac3","Type":"ContainerStarted","Data":"932bdf6028e7fbdbe07b7c4ee3c4f97dabbc85bfa2fadc3956d2dd16b2c18dce"}
Nov 02 09:49:25.813635 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:25.813164    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86" event={"ID":"fda43351-8185-454d-898c-a5b118892ac3","Type":"ContainerStarted","Data":"51929b67b3f75c3126a93374c36d8b8aea50f904bea4ef494bd1712a43cbf559"}
Nov 02 09:49:25.833542 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:25.833475    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86" podStartSLOduration=1.833454429 podStartE2EDuration="1.833454429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:25.832373155 +0000 UTC m=+842.770659952" watchObservedRunningTime="2026-11-02 09:49:25.833454429 +0000 UTC m=+842.771741206"
Nov 02 09:49:26.823271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:26.823229    4449 generic.go:356] "Generic (PLEG): container finished" podID="fda43351-8185-454d-898c-a5b118892ac3" containerID="932bdf6028e7fbdbe07b7c4ee3c4f97dabbc85bfa2fadc3956d2dd16b2c18dce" exitCode=0
Nov 02 09:49:26.823688 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:26.823313    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86" event={"ID":"fda43351-8185-454d-898c-a5b118892ac3","Type":"ContainerDied","Data":"932bdf6028e7fbdbe07b7c4ee3c4f97dabbc85bfa2fadc3956d2dd16b2c18dce"}
Nov 02 09:49:26.903159 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:26.903116    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"]
Nov 02 09:49:26.907798 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:26.907776    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"
Nov 02 09:49:26.916650 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:26.916621    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"]
Nov 02 09:49:27.027057 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:27.026994    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68f86308-h7mvx\" (UID: \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"
Nov 02 09:49:27.027315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:27.027202    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6v7c\" (UniqueName: \"kubernetes.io/projected/67fd243f-cdb4-490e-b83f-41cbbcf209f5-kube-api-access-n6v7c\") pod \"maas-api-revoke-sub-68f86308-h7mvx\" (UID: \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"
Nov 02 09:49:27.027398 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:27.027326    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68f86308-h7mvx\" (UID: \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"
Nov 02 09:49:27.129191 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:27.129101    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6v7c\" (UniqueName: \"kubernetes.io/projected/67fd243f-cdb4-490e-b83f-41cbbcf209f5-kube-api-access-n6v7c\") pod \"maas-api-revoke-sub-68f86308-h7mvx\" (UID: \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"
Nov 02 09:49:27.129191 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:27.129181    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68f86308-h7mvx\" (UID: \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"
Nov 02 09:49:27.129477 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:27.129307    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68f86308-h7mvx\" (UID: \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"
Nov 02 09:49:27.129863 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:27.129839    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68f86308-h7mvx\" (UID: \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"
Nov 02 09:49:27.134241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:27.133627    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68f86308-h7mvx\" (UID: \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"
Nov 02 09:49:27.143093 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:27.143054    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6v7c\" (UniqueName: \"kubernetes.io/projected/67fd243f-cdb4-490e-b83f-41cbbcf209f5-kube-api-access-n6v7c\") pod \"maas-api-revoke-sub-68f86308-h7mvx\" (UID: \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"
Nov 02 09:49:27.230658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:27.230625    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"
Nov 02 09:49:27.560383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:27.560333    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"]
Nov 02 09:49:27.565476 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:49:27.565447    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fd243f_cdb4_490e_b83f_41cbbcf209f5.slice/crio-1f3c2f072c87241cd0f5c9ddd1cfd6333643f4713255929ff2a84fc257798aac WatchSource:0}: Error finding container 1f3c2f072c87241cd0f5c9ddd1cfd6333643f4713255929ff2a84fc257798aac: Status 404 returned error can't find the container with id 1f3c2f072c87241cd0f5c9ddd1cfd6333643f4713255929ff2a84fc257798aac
Nov 02 09:49:27.834711 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:27.834621    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx" event={"ID":"67fd243f-cdb4-490e-b83f-41cbbcf209f5","Type":"ContainerStarted","Data":"1f3c2f072c87241cd0f5c9ddd1cfd6333643f4713255929ff2a84fc257798aac"}
Nov 02 09:49:28.012957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.012921    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"
Nov 02 09:49:28.054391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.054348    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fda43351-8185-454d-898c-a5b118892ac3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda43351-8185-454d-898c-a5b118892ac3-maas-api-xks-ca\") pod \"fda43351-8185-454d-898c-a5b118892ac3\" (UID: \"fda43351-8185-454d-898c-a5b118892ac3\") "
Nov 02 09:49:28.054690 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.054407    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda43351-8185-454d-898c-a5b118892ac3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda43351-8185-454d-898c-a5b118892ac3-maas-api-openshift-ca\") pod \"fda43351-8185-454d-898c-a5b118892ac3\" (UID: \"fda43351-8185-454d-898c-a5b118892ac3\") "
Nov 02 09:49:28.054690 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.054446    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda43351-8185-454d-898c-a5b118892ac3-kube-api-access-gsl9p\" (UniqueName: \"kubernetes.io/projected/fda43351-8185-454d-898c-a5b118892ac3-kube-api-access-gsl9p\") pod \"fda43351-8185-454d-898c-a5b118892ac3\" (UID: \"fda43351-8185-454d-898c-a5b118892ac3\") "
Nov 02 09:49:28.055675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.055625    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda43351-8185-454d-898c-a5b118892ac3-maas-api-openshift-ca" pod "fda43351-8185-454d-898c-a5b118892ac3" (UID: "fda43351-8185-454d-898c-a5b118892ac3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:28.061846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.061799    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda43351-8185-454d-898c-a5b118892ac3-kube-api-access-gsl9p" pod "fda43351-8185-454d-898c-a5b118892ac3" (UID: "fda43351-8185-454d-898c-a5b118892ac3"). InnerVolumeSpecName "kube-api-access-gsl9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:28.063701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.063665    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda43351-8185-454d-898c-a5b118892ac3-maas-api-xks-ca" pod "fda43351-8185-454d-898c-a5b118892ac3" (UID: "fda43351-8185-454d-898c-a5b118892ac3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:49:28.159376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.159327    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda43351-8185-454d-898c-a5b118892ac3-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:28.159376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.159376    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda43351-8185-454d-898c-a5b118892ac3-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:28.159639 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.159392    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsl9p\" (UniqueName: \"kubernetes.io/projected/fda43351-8185-454d-898c-a5b118892ac3-kube-api-access-gsl9p\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:28.845797 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.845764    4449 generic.go:356] "Generic (PLEG): container finished" podID="67fd243f-cdb4-490e-b83f-41cbbcf209f5" containerID="f896790bee3f6ccc535c4b12fa42fbce219b8bf1619422fd7ded5ac7b6ada115" exitCode=0
Nov 02 09:49:28.846287 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.845859    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx" event={"ID":"67fd243f-cdb4-490e-b83f-41cbbcf209f5","Type":"ContainerDied","Data":"f896790bee3f6ccc535c4b12fa42fbce219b8bf1619422fd7ded5ac7b6ada115"}
Nov 02 09:49:28.847631 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.847605    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"
Nov 02 09:49:28.847883 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.847640    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86" event={"ID":"fda43351-8185-454d-898c-a5b118892ac3","Type":"ContainerDied","Data":"51929b67b3f75c3126a93374c36d8b8aea50f904bea4ef494bd1712a43cbf559"}
Nov 02 09:49:28.847997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:28.847905    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51929b67b3f75c3126a93374c36d8b8aea50f904bea4ef494bd1712a43cbf559"
Nov 02 09:49:30.056363 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:30.056334    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"
Nov 02 09:49:30.093984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:30.093939    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67fd243f-cdb4-490e-b83f-41cbbcf209f5-kube-api-access-n6v7c\" (UniqueName: \"kubernetes.io/projected/67fd243f-cdb4-490e-b83f-41cbbcf209f5-kube-api-access-n6v7c\") pod \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\" (UID: \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\") "
Nov 02 09:49:30.094317 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:30.094292    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-openshift-ca\") pod \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\" (UID: \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\") "
Nov 02 09:49:30.094419 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:30.094356    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-xks-ca\") pod \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\" (UID: \"67fd243f-cdb4-490e-b83f-41cbbcf209f5\") "
Nov 02 09:49:30.094782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:30.094754    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-openshift-ca" pod "67fd243f-cdb4-490e-b83f-41cbbcf209f5" (UID: "67fd243f-cdb4-490e-b83f-41cbbcf209f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:30.094960 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:30.094887    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:30.096544 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:30.096513    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fd243f-cdb4-490e-b83f-41cbbcf209f5-kube-api-access-n6v7c" pod "67fd243f-cdb4-490e-b83f-41cbbcf209f5" (UID: "67fd243f-cdb4-490e-b83f-41cbbcf209f5"). InnerVolumeSpecName "kube-api-access-n6v7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:30.097646 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:30.097614    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-xks-ca" pod "67fd243f-cdb4-490e-b83f-41cbbcf209f5" (UID: "67fd243f-cdb4-490e-b83f-41cbbcf209f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:49:30.195812 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:30.195773    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fd243f-cdb4-490e-b83f-41cbbcf209f5-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:30.195812 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:30.195810    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6v7c\" (UniqueName: \"kubernetes.io/projected/67fd243f-cdb4-490e-b83f-41cbbcf209f5-kube-api-access-n6v7c\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:30.865638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:30.865608    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"
Nov 02 09:49:30.865831 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:30.865637    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx" event={"ID":"67fd243f-cdb4-490e-b83f-41cbbcf209f5","Type":"ContainerDied","Data":"1f3c2f072c87241cd0f5c9ddd1cfd6333643f4713255929ff2a84fc257798aac"}
Nov 02 09:49:30.865831 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:30.865727    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f3c2f072c87241cd0f5c9ddd1cfd6333643f4713255929ff2a84fc257798aac"
Nov 02 09:49:33.098712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.098674    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"]
Nov 02 09:49:33.134245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.109413    4449 cpu_manager.go:414] "containerMap: removing container" podUID="67fd243f-cdb4-490e-b83f-41cbbcf209f5" containerName="revoke-keys"
Nov 02 09:49:33.134603 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.134515    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67fd243f-cdb4-490e-b83f-41cbbcf209f5" containerName="revoke-keys"
Nov 02 09:49:33.134847 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.134813    4449 cpu_manager.go:414] "containerMap: removing container" podUID="fda43351-8185-454d-898c-a5b118892ac3" containerName="revoke-keys"
Nov 02 09:49:33.134975 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.134961    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda43351-8185-454d-898c-a5b118892ac3" containerName="revoke-keys"
Nov 02 09:49:33.135435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.135416    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="67fd243f-cdb4-490e-b83f-41cbbcf209f5" containerName="revoke-keys"
Nov 02 09:49:33.135589 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.135576    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda43351-8185-454d-898c-a5b118892ac3" containerName="revoke-keys"
Nov 02 09:49:33.530308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.529991    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.530308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.530110    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.530308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.530177    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.530308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.530208    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.530308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.530291    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978d530c-fd42-4e42-80cd-746b1e37f40f-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.530769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.530359    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.530769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.530443    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.530769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.530522    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.530769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.530565    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9blh\" (UniqueName: \"kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-kube-api-access-d9blh\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.597246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.587569    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"]
Nov 02 09:49:33.597246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.593349    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.620245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.619550    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-204fba-openshift-default-dockercfg-xvjxq\""
Nov 02 09:49:33.639022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.634544    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.639022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.634626    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.639022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.634660    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.639022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.634722    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978d530c-fd42-4e42-80cd-746b1e37f40f-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.639022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.634786    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.639022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.634866    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.639022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.634945    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.639022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.634986    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9blh\" (UniqueName: \"kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-kube-api-access-d9blh\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.639022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.635030    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.639022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.635663    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.639022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.635916    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.639022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.636156    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.687246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.668363    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978d530c-fd42-4e42-80cd-746b1e37f40f-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.687246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.669016    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.687246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.670545    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.708306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.699566    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.766273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.749072    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:33.874570 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:33.860497    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9blh\" (UniqueName: \"kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-kube-api-access-d9blh\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:34.129389 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:34.116752    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:34.366368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:34.366309    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" podUID="e5827546-705c-4b91-a21e-daacfee4134a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Nov 02 09:49:34.537309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:34.534569    4449 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz" podUID="5ed165dd-1500-4fd4-a5ae-5a35d894726e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:34.717323 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:34.715483    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:34.779258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:34.772676    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:49:34.779258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:34.772782    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:34.795163 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:34.795115    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:34.854903 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:34.846512    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz" podUID="5ed165dd-1500-4fd4-a5ae-5a35d894726e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:34.978325 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:34.974634    4449 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:35.035685 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.035627    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:35.071260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.068364    4449 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:49:35.071260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.068471    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:35.399410 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.398784    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" podUID="e5827546-705c-4b91-a21e-daacfee4134a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:35.454438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.454400    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:49:35.454675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.454657    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"]
Nov 02 09:49:35.480834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.480794    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"]
Nov 02 09:49:35.489267 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.481115    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"
Nov 02 09:49:35.547327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.546478    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:49:35.547327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.546725    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:49:35.547327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.546790    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:49:35.547327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.546875    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:49:35.547867 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.547833    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:49:35.650912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.649282    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dc58f74-2s5hz\" (UID: \"5f5b50ce-7305-4667-bb40-0b3cad698096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"
Nov 02 09:49:35.650912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.649552    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dc58f74-2s5hz\" (UID: \"5f5b50ce-7305-4667-bb40-0b3cad698096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"
Nov 02 09:49:35.650912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.649598    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzqr9\" (UniqueName: \"kubernetes.io/projected/5f5b50ce-7305-4667-bb40-0b3cad698096-kube-api-access-dzqr9\") pod \"maas-api-revoke-sub-8dc58f74-2s5hz\" (UID: \"5f5b50ce-7305-4667-bb40-0b3cad698096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"
Nov 02 09:49:35.745882 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:49:35.745849    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978d530c_fd42_4e42_80cd_746b1e37f40f.slice/crio-ccd128c50a4bc0b2349d0248216b1e8f8446f4fb200a6552739c5040fd808247 WatchSource:0}: Error finding container ccd128c50a4bc0b2349d0248216b1e8f8446f4fb200a6552739c5040fd808247: Status 404 returned error can't find the container with id ccd128c50a4bc0b2349d0248216b1e8f8446f4fb200a6552739c5040fd808247
Nov 02 09:49:35.746999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.746968    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"]
Nov 02 09:49:35.749389 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.749341    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:49:35.749509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.749443    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:49:35.749509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.749494    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:49:35.750696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.750671    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dc58f74-2s5hz\" (UID: \"5f5b50ce-7305-4667-bb40-0b3cad698096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"
Nov 02 09:49:35.750974 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.750932    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dc58f74-2s5hz\" (UID: \"5f5b50ce-7305-4667-bb40-0b3cad698096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"
Nov 02 09:49:35.751079 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.750985    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzqr9\" (UniqueName: \"kubernetes.io/projected/5f5b50ce-7305-4667-bb40-0b3cad698096-kube-api-access-dzqr9\") pod \"maas-api-revoke-sub-8dc58f74-2s5hz\" (UID: \"5f5b50ce-7305-4667-bb40-0b3cad698096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"
Nov 02 09:49:35.751790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.751764    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dc58f74-2s5hz\" (UID: \"5f5b50ce-7305-4667-bb40-0b3cad698096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"
Nov 02 09:49:35.754670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.754644    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dc58f74-2s5hz\" (UID: \"5f5b50ce-7305-4667-bb40-0b3cad698096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"
Nov 02 09:49:35.765291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.765259    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzqr9\" (UniqueName: \"kubernetes.io/projected/5f5b50ce-7305-4667-bb40-0b3cad698096-kube-api-access-dzqr9\") pod \"maas-api-revoke-sub-8dc58f74-2s5hz\" (UID: \"5f5b50ce-7305-4667-bb40-0b3cad698096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"
Nov 02 09:49:35.891276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:35.891243    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"
Nov 02 09:49:36.058316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:36.058270    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9" event={"ID":"978d530c-fd42-4e42-80cd-746b1e37f40f","Type":"ContainerStarted","Data":"6f2d2417ddd9aa48edb2315aa07f0ed216def28593afbaa677f3d7c4d3421f22"}
Nov 02 09:49:36.058518 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:36.058345    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9" event={"ID":"978d530c-fd42-4e42-80cd-746b1e37f40f","Type":"ContainerStarted","Data":"ccd128c50a4bc0b2349d0248216b1e8f8446f4fb200a6552739c5040fd808247"}
Nov 02 09:49:36.085982 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:36.085860    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9" podStartSLOduration=3.085841733 podStartE2EDuration="3.085841733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:36.082512233 +0000 UTC m=+853.020799034" watchObservedRunningTime="2026-11-02 09:49:36.085841733 +0000 UTC m=+853.024128510"
Nov 02 09:49:36.117596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:36.117563    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:36.119193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:36.119166    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 09:49:36.119357 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:36.119253    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9" podUID="978d530c-fd42-4e42-80cd-746b1e37f40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 09:49:36.195687 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:49:36.195643    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5b50ce_7305_4667_bb40_0b3cad698096.slice/crio-7899b005d4d9d16d9e4e28f02c2ba4b42d12493b38000288f14729174201d295 WatchSource:0}: Error finding container 7899b005d4d9d16d9e4e28f02c2ba4b42d12493b38000288f14729174201d295: Status 404 returned error can't find the container with id 7899b005d4d9d16d9e4e28f02c2ba4b42d12493b38000288f14729174201d295
Nov 02 09:49:36.199795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:36.199765    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"]
Nov 02 09:49:36.574511 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:49:36.574441    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:49:37.084133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:37.084096    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz" event={"ID":"5f5b50ce-7305-4667-bb40-0b3cad698096","Type":"ContainerStarted","Data":"6b1a78e10b68330aa65d7d0aad18f38c185be0ff63452c9e4b23cc8978300e63"}
Nov 02 09:49:37.084357 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:37.084149    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz" event={"ID":"5f5b50ce-7305-4667-bb40-0b3cad698096","Type":"ContainerStarted","Data":"7899b005d4d9d16d9e4e28f02c2ba4b42d12493b38000288f14729174201d295"}
Nov 02 09:49:37.110415 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:37.110367    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz" podStartSLOduration=2.110353694 podStartE2EDuration="2.110353694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:37.106303515 +0000 UTC m=+854.044590315" watchObservedRunningTime="2026-11-02 09:49:37.110353694 +0000 UTC m=+854.048640469"
Nov 02 09:49:37.117691 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:37.117661    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 09:49:37.117834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:37.117721    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9" podUID="978d530c-fd42-4e42-80cd-746b1e37f40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 09:49:38.094145 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:38.094117    4449 generic.go:356] "Generic (PLEG): container finished" podID="5f5b50ce-7305-4667-bb40-0b3cad698096" containerID="6b1a78e10b68330aa65d7d0aad18f38c185be0ff63452c9e4b23cc8978300e63" exitCode=0
Nov 02 09:49:38.094616 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:38.094203    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz" event={"ID":"5f5b50ce-7305-4667-bb40-0b3cad698096","Type":"ContainerDied","Data":"6b1a78e10b68330aa65d7d0aad18f38c185be0ff63452c9e4b23cc8978300e63"}
Nov 02 09:49:38.117681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:38.117644    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 09:49:38.117838 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:38.117704    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9" podUID="978d530c-fd42-4e42-80cd-746b1e37f40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 09:49:39.117533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:39.117495    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 09:49:39.118183 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:39.117551    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9" podUID="978d530c-fd42-4e42-80cd-746b1e37f40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 09:49:39.311986 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:39.311958    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"
Nov 02 09:49:39.412816 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:39.412725    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-xks-ca\") pod \"5f5b50ce-7305-4667-bb40-0b3cad698096\" (UID: \"5f5b50ce-7305-4667-bb40-0b3cad698096\") "
Nov 02 09:49:39.412816 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:39.412774    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5b50ce-7305-4667-bb40-0b3cad698096-kube-api-access-dzqr9\" (UniqueName: \"kubernetes.io/projected/5f5b50ce-7305-4667-bb40-0b3cad698096-kube-api-access-dzqr9\") pod \"5f5b50ce-7305-4667-bb40-0b3cad698096\" (UID: \"5f5b50ce-7305-4667-bb40-0b3cad698096\") "
Nov 02 09:49:39.412816 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:39.412817    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-openshift-ca\") pod \"5f5b50ce-7305-4667-bb40-0b3cad698096\" (UID: \"5f5b50ce-7305-4667-bb40-0b3cad698096\") "
Nov 02 09:49:39.413509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:39.413478    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-openshift-ca" pod "5f5b50ce-7305-4667-bb40-0b3cad698096" (UID: "5f5b50ce-7305-4667-bb40-0b3cad698096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:39.414993 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:39.414970    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-xks-ca" pod "5f5b50ce-7305-4667-bb40-0b3cad698096" (UID: "5f5b50ce-7305-4667-bb40-0b3cad698096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:49:39.415122 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:39.415104    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5b50ce-7305-4667-bb40-0b3cad698096-kube-api-access-dzqr9" pod "5f5b50ce-7305-4667-bb40-0b3cad698096" (UID: "5f5b50ce-7305-4667-bb40-0b3cad698096"). InnerVolumeSpecName "kube-api-access-dzqr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:39.516043 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:39.515987    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:39.516043 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:39.516038    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzqr9\" (UniqueName: \"kubernetes.io/projected/5f5b50ce-7305-4667-bb40-0b3cad698096-kube-api-access-dzqr9\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:39.516356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:39.516057    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5b50ce-7305-4667-bb40-0b3cad698096-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:40.113056 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:40.113020    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"
Nov 02 09:49:40.113272 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:40.113060    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz" event={"ID":"5f5b50ce-7305-4667-bb40-0b3cad698096","Type":"ContainerDied","Data":"7899b005d4d9d16d9e4e28f02c2ba4b42d12493b38000288f14729174201d295"}
Nov 02 09:49:40.113272 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:40.113115    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7899b005d4d9d16d9e4e28f02c2ba4b42d12493b38000288f14729174201d295"
Nov 02 09:49:40.117575 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:40.117546    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 09:49:40.117927 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:40.117607    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9" podUID="978d530c-fd42-4e42-80cd-746b1e37f40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 09:49:41.118682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:41.118647    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 09:49:41.119082 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:41.118709    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9" podUID="978d530c-fd42-4e42-80cd-746b1e37f40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 09:49:42.118323 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:42.118276    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 09:49:42.118537 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:42.118351    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9" podUID="978d530c-fd42-4e42-80cd-746b1e37f40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 09:49:43.122742 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:43.122713    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:43.123254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:43.123207    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:43.123948 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:43.123932    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 09:49:44.040415 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:44.040376    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"]
Nov 02 09:49:44.047728 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:44.047702    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a32d6aa-67s9z"]
Nov 02 09:49:45.211254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:45.211193    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="180a67b5-4edb-4572-b71c-5913c10a34e3" path="/var/lib/kubelet/pods/180a67b5-4edb-4572-b71c-5913c10a34e3/volumes"
Nov 02 09:49:48.009436 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:48.009370    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:48.733248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:48.702315    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:48.889264 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:48.846824    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz" podUID="5ed165dd-1500-4fd4-a5ae-5a35d894726e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 02 09:49:49.194289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:49.178287    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:49.776241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:49.775191    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"]
Nov 02 09:49:50.020993 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.020908    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"]
Nov 02 09:49:50.022755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.022726    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5f5b50ce-7305-4667-bb40-0b3cad698096" containerName="revoke-keys"
Nov 02 09:49:50.022934 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.022920    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5b50ce-7305-4667-bb40-0b3cad698096" containerName="revoke-keys"
Nov 02 09:49:50.023433 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.023414    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5b50ce-7305-4667-bb40-0b3cad698096" containerName="revoke-keys"
Nov 02 09:49:50.031801 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.031734    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"]
Nov 02 09:49:50.031903 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.031749    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"
Nov 02 09:49:50.041250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.040416    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:49:50.041250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.040583    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:49:50.057538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.057510    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:49:50.057844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.057823    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:49:50.171708 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.171088    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c433d6fa-pggx5\" (UID: \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"
Nov 02 09:49:50.171708 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.171173    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c433d6fa-pggx5\" (UID: \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"
Nov 02 09:49:50.171708 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.171318    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp5hn\" (UniqueName: \"kubernetes.io/projected/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-kube-api-access-cp5hn\") pod \"maas-api-revoke-sub-c433d6fa-pggx5\" (UID: \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"
Nov 02 09:49:50.272942 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.272898    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c433d6fa-pggx5\" (UID: \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"
Nov 02 09:49:50.273488 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.273057    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp5hn\" (UniqueName: \"kubernetes.io/projected/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-kube-api-access-cp5hn\") pod \"maas-api-revoke-sub-c433d6fa-pggx5\" (UID: \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"
Nov 02 09:49:50.273488 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.273256    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c433d6fa-pggx5\" (UID: \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"
Nov 02 09:49:50.273821 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.273788    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c433d6fa-pggx5\" (UID: \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"
Nov 02 09:49:50.288684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.288624    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c433d6fa-pggx5\" (UID: \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"
Nov 02 09:49:50.291649 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.291616    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp5hn\" (UniqueName: \"kubernetes.io/projected/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-kube-api-access-cp5hn\") pod \"maas-api-revoke-sub-c433d6fa-pggx5\" (UID: \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"
Nov 02 09:49:50.369667 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.369632    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"
Nov 02 09:49:50.685948 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:50.684173    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"]
Nov 02 09:49:51.470019 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:51.469517    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5" event={"ID":"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800","Type":"ContainerStarted","Data":"ab9ed78d7ac61f39e633a17f875f84269710412052a0863fee7f6ba1338bf0e4"}
Nov 02 09:49:51.470568 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:51.470031    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5" event={"ID":"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800","Type":"ContainerStarted","Data":"cec085925be74afc6ce0447b6ec6d52cad093e6dc87054325e6d8c81c36f5560"}
Nov 02 09:49:51.494374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:51.494310    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5" podStartSLOduration=2.494285999 podStartE2EDuration="2.494285999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:51.491092415 +0000 UTC m=+868.429379208" watchObservedRunningTime="2026-11-02 09:49:51.494285999 +0000 UTC m=+868.432572780"
Nov 02 09:49:52.259241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.259113    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"]
Nov 02 09:49:52.289261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.289188    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"]
Nov 02 09:49:52.289543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.289364    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"
Nov 02 09:49:52.427436 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.427395    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kh58\" (UniqueName: \"kubernetes.io/projected/9366ff10-7d62-482f-a721-e626f53cf72d-kube-api-access-2kh58\") pod \"maas-api-revoke-sub-391e80b2-9czlt\" (UID: \"9366ff10-7d62-482f-a721-e626f53cf72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"
Nov 02 09:49:52.427627 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.427566    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-391e80b2-9czlt\" (UID: \"9366ff10-7d62-482f-a721-e626f53cf72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"
Nov 02 09:49:52.427627 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.427608    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-391e80b2-9czlt\" (UID: \"9366ff10-7d62-482f-a721-e626f53cf72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"
Nov 02 09:49:52.479819 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.479785    4449 generic.go:356] "Generic (PLEG): container finished" podID="0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800" containerID="ab9ed78d7ac61f39e633a17f875f84269710412052a0863fee7f6ba1338bf0e4" exitCode=0
Nov 02 09:49:52.480279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.479874    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5" event={"ID":"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800","Type":"ContainerDied","Data":"ab9ed78d7ac61f39e633a17f875f84269710412052a0863fee7f6ba1338bf0e4"}
Nov 02 09:49:52.529004 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.528912    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kh58\" (UniqueName: \"kubernetes.io/projected/9366ff10-7d62-482f-a721-e626f53cf72d-kube-api-access-2kh58\") pod \"maas-api-revoke-sub-391e80b2-9czlt\" (UID: \"9366ff10-7d62-482f-a721-e626f53cf72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"
Nov 02 09:49:52.529193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.529038    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-391e80b2-9czlt\" (UID: \"9366ff10-7d62-482f-a721-e626f53cf72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"
Nov 02 09:49:52.529193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.529069    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-391e80b2-9czlt\" (UID: \"9366ff10-7d62-482f-a721-e626f53cf72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"
Nov 02 09:49:52.529914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.529886    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-391e80b2-9czlt\" (UID: \"9366ff10-7d62-482f-a721-e626f53cf72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"
Nov 02 09:49:52.531656 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.531624    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-391e80b2-9czlt\" (UID: \"9366ff10-7d62-482f-a721-e626f53cf72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"
Nov 02 09:49:52.561002 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.560973    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kh58\" (UniqueName: \"kubernetes.io/projected/9366ff10-7d62-482f-a721-e626f53cf72d-kube-api-access-2kh58\") pod \"maas-api-revoke-sub-391e80b2-9czlt\" (UID: \"9366ff10-7d62-482f-a721-e626f53cf72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"
Nov 02 09:49:52.646437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.646405    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"
Nov 02 09:49:52.935265 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:49:52.935230    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9366ff10_7d62_482f_a721_e626f53cf72d.slice/crio-975d3e99e46d6bc82897b40639d84ce2974fdc61a52db990b6e8f1d26393f1ee WatchSource:0}: Error finding container 975d3e99e46d6bc82897b40639d84ce2974fdc61a52db990b6e8f1d26393f1ee: Status 404 returned error can't find the container with id 975d3e99e46d6bc82897b40639d84ce2974fdc61a52db990b6e8f1d26393f1ee
Nov 02 09:49:52.937824 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:52.937796    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"]
Nov 02 09:49:53.553884 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:53.553841    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt" event={"ID":"9366ff10-7d62-482f-a721-e626f53cf72d","Type":"ContainerStarted","Data":"7b95233ee9918fdb7ef4ad32ed0f79280b5ce4fd9fe95b6a24201839f54a4247"}
Nov 02 09:49:53.554459 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:53.554414    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt" event={"ID":"9366ff10-7d62-482f-a721-e626f53cf72d","Type":"ContainerStarted","Data":"975d3e99e46d6bc82897b40639d84ce2974fdc61a52db990b6e8f1d26393f1ee"}
Nov 02 09:49:53.616496 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:53.609341    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt" podStartSLOduration=1.609319891 podStartE2EDuration="1.609319891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:53.596556931 +0000 UTC m=+870.534843724" watchObservedRunningTime="2026-11-02 09:49:53.609319891 +0000 UTC m=+870.547606661"
Nov 02 09:49:53.962692 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:53.962662    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"
Nov 02 09:49:54.097057 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.097024    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-kube-api-access-cp5hn\" (UniqueName: \"kubernetes.io/projected/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-kube-api-access-cp5hn\") pod \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\" (UID: \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\") "
Nov 02 09:49:54.097338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.097182    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-openshift-ca\") pod \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\" (UID: \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\") "
Nov 02 09:49:54.097338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.097209    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-xks-ca\") pod \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\" (UID: \"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800\") "
Nov 02 09:49:54.097681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.097656    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-openshift-ca" pod "0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800" (UID: "0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:54.098037 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.098009    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:54.099526 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.099502    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-kube-api-access-cp5hn" pod "0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800" (UID: "0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800"). InnerVolumeSpecName "kube-api-access-cp5hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:54.099813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.099770    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-xks-ca" pod "0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800" (UID: "0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:49:54.198901 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.198823    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:54.198901 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.198850    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp5hn\" (UniqueName: \"kubernetes.io/projected/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800-kube-api-access-cp5hn\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:54.553805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.553770    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"
Nov 02 09:49:54.554503 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.554475    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5" event={"ID":"0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800","Type":"ContainerDied","Data":"cec085925be74afc6ce0447b6ec6d52cad093e6dc87054325e6d8c81c36f5560"}
Nov 02 09:49:54.554926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.554534    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cec085925be74afc6ce0447b6ec6d52cad093e6dc87054325e6d8c81c36f5560"
Nov 02 09:49:54.556616 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.556591    4449 generic.go:356] "Generic (PLEG): container finished" podID="9366ff10-7d62-482f-a721-e626f53cf72d" containerID="7b95233ee9918fdb7ef4ad32ed0f79280b5ce4fd9fe95b6a24201839f54a4247" exitCode=0
Nov 02 09:49:54.556746 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:54.556668    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt" event={"ID":"9366ff10-7d62-482f-a721-e626f53cf72d","Type":"ContainerDied","Data":"7b95233ee9918fdb7ef4ad32ed0f79280b5ce4fd9fe95b6a24201839f54a4247"}
Nov 02 09:49:55.758544 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:55.758520    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"
Nov 02 09:49:55.821017 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:55.820934    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-xks-ca\") pod \"9366ff10-7d62-482f-a721-e626f53cf72d\" (UID: \"9366ff10-7d62-482f-a721-e626f53cf72d\") "
Nov 02 09:49:55.821017 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:55.821010    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-openshift-ca\") pod \"9366ff10-7d62-482f-a721-e626f53cf72d\" (UID: \"9366ff10-7d62-482f-a721-e626f53cf72d\") "
Nov 02 09:49:55.821274 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:55.821160    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9366ff10-7d62-482f-a721-e626f53cf72d-kube-api-access-2kh58\" (UniqueName: \"kubernetes.io/projected/9366ff10-7d62-482f-a721-e626f53cf72d-kube-api-access-2kh58\") pod \"9366ff10-7d62-482f-a721-e626f53cf72d\" (UID: \"9366ff10-7d62-482f-a721-e626f53cf72d\") "
Nov 02 09:49:55.821467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:55.821436    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-openshift-ca" pod "9366ff10-7d62-482f-a721-e626f53cf72d" (UID: "9366ff10-7d62-482f-a721-e626f53cf72d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:55.821877 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:55.821858    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:55.823380 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:55.823358    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-xks-ca" pod "9366ff10-7d62-482f-a721-e626f53cf72d" (UID: "9366ff10-7d62-482f-a721-e626f53cf72d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:49:55.823887 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:55.823870    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9366ff10-7d62-482f-a721-e626f53cf72d-kube-api-access-2kh58" pod "9366ff10-7d62-482f-a721-e626f53cf72d" (UID: "9366ff10-7d62-482f-a721-e626f53cf72d"). InnerVolumeSpecName "kube-api-access-2kh58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:55.923128 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:55.923099    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kh58\" (UniqueName: \"kubernetes.io/projected/9366ff10-7d62-482f-a721-e626f53cf72d-kube-api-access-2kh58\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:55.923128 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:55.923127    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9366ff10-7d62-482f-a721-e626f53cf72d-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:56.574485 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:56.574456    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"
Nov 02 09:49:56.574660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:56.574496    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt" event={"ID":"9366ff10-7d62-482f-a721-e626f53cf72d","Type":"ContainerDied","Data":"975d3e99e46d6bc82897b40639d84ce2974fdc61a52db990b6e8f1d26393f1ee"}
Nov 02 09:49:56.574660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:56.574546    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="975d3e99e46d6bc82897b40639d84ce2974fdc61a52db990b6e8f1d26393f1ee"
Nov 02 09:49:59.452755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.452717    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"]
Nov 02 09:49:59.453789 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.453766    4449 cpu_manager.go:414] "containerMap: removing container" podUID="9366ff10-7d62-482f-a721-e626f53cf72d" containerName="revoke-keys"
Nov 02 09:49:59.453915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.453792    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9366ff10-7d62-482f-a721-e626f53cf72d" containerName="revoke-keys"
Nov 02 09:49:59.453915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.453829    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800" containerName="revoke-keys"
Nov 02 09:49:59.453915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.453838    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800" containerName="revoke-keys"
Nov 02 09:49:59.454079 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.454039    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="9366ff10-7d62-482f-a721-e626f53cf72d" containerName="revoke-keys"
Nov 02 09:49:59.454079 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.454056    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800" containerName="revoke-keys"
Nov 02 09:49:59.458813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.458791    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.466313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.466277    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2e80fb\"/\"distinct-2-w0-2e80fb-kserve-self-signed-certs\""
Nov 02 09:49:59.467378 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.467352    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2e80fb\"/\"default-dockercfg-gkpxc\""
Nov 02 09:49:59.467544 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.467517    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"]
Nov 02 09:49:59.469113 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.469091    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2e80fb\"/\"openshift-service-ca.crt\""
Nov 02 09:49:59.469326 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.469308    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2e80fb\"/\"kube-root-ca.crt\""
Nov 02 09:49:59.573087 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.572998    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-dshm\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.573087 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.573049    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-home\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.573087 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.573079    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5l57\" (UniqueName: \"kubernetes.io/projected/45450305-a85b-470b-b163-b4a2d35a3cd0-kube-api-access-h5l57\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.573813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.573307    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-model-cache\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.573813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.573339    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-tmp-dir\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.573813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.573399    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45450305-a85b-470b-b163-b4a2d35a3cd0-tls-certs\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.675241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.675188    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-model-cache\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.675241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.675243    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-tmp-dir\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.675507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.675289    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45450305-a85b-470b-b163-b4a2d35a3cd0-tls-certs\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.675507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.675371    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-dshm\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.675507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.675389    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-home\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.675507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.675404    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5l57\" (UniqueName: \"kubernetes.io/projected/45450305-a85b-470b-b163-b4a2d35a3cd0-kube-api-access-h5l57\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.675720 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.675653    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-model-cache\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.675720 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.675697    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-home\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.675837 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.675814    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-tmp-dir\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.678355 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.678330    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-dshm\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.678550 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.678302    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45450305-a85b-470b-b163-b4a2d35a3cd0-tls-certs\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.690490 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.690451    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5l57\" (UniqueName: \"kubernetes.io/projected/45450305-a85b-470b-b163-b4a2d35a3cd0-kube-api-access-h5l57\") pod \"distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:49:59.774600 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:49:59.774507    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:50:00.068373 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:00.068342    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45450305_a85b_470b_b163_b4a2d35a3cd0.slice/crio-3483b76339fe787b1469c7e9e20eaa875376598e4a1d0ddb250abcad1fb690c1 WatchSource:0}: Error finding container 3483b76339fe787b1469c7e9e20eaa875376598e4a1d0ddb250abcad1fb690c1: Status 404 returned error can't find the container with id 3483b76339fe787b1469c7e9e20eaa875376598e4a1d0ddb250abcad1fb690c1
Nov 02 09:50:00.068676 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:00.068654    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"]
Nov 02 09:50:00.643580 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:00.643533    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" event={"ID":"45450305-a85b-470b-b163-b4a2d35a3cd0","Type":"ContainerStarted","Data":"a9fc3710565f6b71f2d5a53c180169afe6c47eb2e10d5c73e9044ea33267546f"}
Nov 02 09:50:00.644130 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:00.643597    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" event={"ID":"45450305-a85b-470b-b163-b4a2d35a3cd0","Type":"ContainerStarted","Data":"3483b76339fe787b1469c7e9e20eaa875376598e4a1d0ddb250abcad1fb690c1"}
Nov 02 09:50:00.669615 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:00.669551    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podStartSLOduration=1.669529292 podStartE2EDuration="1.669529292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:00.662834193 +0000 UTC m=+877.601120992" watchObservedRunningTime="2026-11-02 09:50:00.669529292 +0000 UTC m=+877.607816069"
Nov 02 09:50:00.771786 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:00.771749    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"]
Nov 02 09:50:00.775257 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:00.775228    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:50:02.731828 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:02.729964    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"]
Nov 02 09:50:02.786377 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:02.786338    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"
Nov 02 09:50:02.820544 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:02.820512    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"]
Nov 02 09:50:02.821803 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:02.821754    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:50:02.831727 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:02.821976    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:50:02.833312 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:02.822074    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:50:02.833602 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:02.830950    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:50:02.900541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:02.900509    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abd856d0-xpfkn\" (UID: \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"
Nov 02 09:50:02.901015 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:02.900990    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abd856d0-xpfkn\" (UID: \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"
Nov 02 09:50:02.901199 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:02.901179    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm667\" (UniqueName: \"kubernetes.io/projected/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-kube-api-access-mm667\") pod \"maas-api-revoke-sub-abd856d0-xpfkn\" (UID: \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"
Nov 02 09:50:03.036304 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:03.031529    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abd856d0-xpfkn\" (UID: \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"
Nov 02 09:50:03.036304 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:03.031838    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abd856d0-xpfkn\" (UID: \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"
Nov 02 09:50:03.062279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:03.060840    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm667\" (UniqueName: \"kubernetes.io/projected/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-kube-api-access-mm667\") pod \"maas-api-revoke-sub-abd856d0-xpfkn\" (UID: \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"
Nov 02 09:50:03.075745 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:03.063351    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abd856d0-xpfkn\" (UID: \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"
Nov 02 09:50:03.141872 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:03.141725    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abd856d0-xpfkn\" (UID: \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"
Nov 02 09:50:03.145249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:03.144292    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm667\" (UniqueName: \"kubernetes.io/projected/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-kube-api-access-mm667\") pod \"maas-api-revoke-sub-abd856d0-xpfkn\" (UID: \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"
Nov 02 09:50:03.234702 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:03.229745    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"
Nov 02 09:50:03.734766 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:03.715615    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"]
Nov 02 09:50:03.822450 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:03.822355    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn" event={"ID":"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257","Type":"ContainerStarted","Data":"4636f3fa268f9a1cc1e559f40bb7b0530ff503a10a9427586f5fb1474f6725b9"}
Nov 02 09:50:04.832744 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:04.832711    4449 generic.go:356] "Generic (PLEG): container finished" podID="e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257" containerID="eb98dcb55a70e698a2c3d3aa47d43ca7b5de9eddf5916b1daf214ad9b161e725" exitCode=0
Nov 02 09:50:04.833167 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:04.832810    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn" event={"ID":"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257","Type":"ContainerDied","Data":"eb98dcb55a70e698a2c3d3aa47d43ca7b5de9eddf5916b1daf214ad9b161e725"}
Nov 02 09:50:04.868677 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:04.868639    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz" podUID="5ed165dd-1500-4fd4-a5ae-5a35d894726e" containerName="main" containerID="cri-o://11822a424089808f6099254a352c407abd819c54e1d747af6d0a87835ebd1ec0" gracePeriod=45
Nov 02 09:50:05.159566 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.159531    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:50:05.368360 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.368324    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-model-cache\") pod \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") "
Nov 02 09:50:05.368564 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.368372    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tls-certs\") pod \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") "
Nov 02 09:50:05.368564 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.368433    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-home\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-home\") pod \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") "
Nov 02 09:50:05.368670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.368579    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tmp-dir\") pod \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") "
Nov 02 09:50:05.368670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.368595    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-model-cache" pod "5ed165dd-1500-4fd4-a5ae-5a35d894726e" (UID: "5ed165dd-1500-4fd4-a5ae-5a35d894726e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:05.368670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.368657    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed165dd-1500-4fd4-a5ae-5a35d894726e-kube-api-access-n895m\" (UniqueName: \"kubernetes.io/projected/5ed165dd-1500-4fd4-a5ae-5a35d894726e-kube-api-access-n895m\") pod \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") "
Nov 02 09:50:05.368833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.368706    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-dshm\") pod \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\" (UID: \"5ed165dd-1500-4fd4-a5ae-5a35d894726e\") "
Nov 02 09:50:05.368833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.368730    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-home" pod "5ed165dd-1500-4fd4-a5ae-5a35d894726e" (UID: "5ed165dd-1500-4fd4-a5ae-5a35d894726e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:05.368926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.368864    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tmp-dir" pod "5ed165dd-1500-4fd4-a5ae-5a35d894726e" (UID: "5ed165dd-1500-4fd4-a5ae-5a35d894726e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:05.369460 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.369378    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:05.369460 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.369401    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:05.369460 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.369411    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:05.370910 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.370883    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-dshm" pod "5ed165dd-1500-4fd4-a5ae-5a35d894726e" (UID: "5ed165dd-1500-4fd4-a5ae-5a35d894726e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:05.371015 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.370941    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tls-certs" pod "5ed165dd-1500-4fd4-a5ae-5a35d894726e" (UID: "5ed165dd-1500-4fd4-a5ae-5a35d894726e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:50:05.371015 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.370949    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed165dd-1500-4fd4-a5ae-5a35d894726e-kube-api-access-n895m" pod "5ed165dd-1500-4fd4-a5ae-5a35d894726e" (UID: "5ed165dd-1500-4fd4-a5ae-5a35d894726e"). InnerVolumeSpecName "kube-api-access-n895m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:05.470775 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.470736    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed165dd-1500-4fd4-a5ae-5a35d894726e-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:05.470775 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.470772    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n895m\" (UniqueName: \"kubernetes.io/projected/5ed165dd-1500-4fd4-a5ae-5a35d894726e-kube-api-access-n895m\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:05.470775 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.470782    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ed165dd-1500-4fd4-a5ae-5a35d894726e-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:05.843309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.843275    4449 generic.go:356] "Generic (PLEG): container finished" podID="5ed165dd-1500-4fd4-a5ae-5a35d894726e" containerID="11822a424089808f6099254a352c407abd819c54e1d747af6d0a87835ebd1ec0" exitCode=0
Nov 02 09:50:05.843777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.843321    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"
Nov 02 09:50:05.843777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.843356    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz" event={"ID":"5ed165dd-1500-4fd4-a5ae-5a35d894726e","Type":"ContainerDied","Data":"11822a424089808f6099254a352c407abd819c54e1d747af6d0a87835ebd1ec0"}
Nov 02 09:50:05.843777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.843394    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz" event={"ID":"5ed165dd-1500-4fd4-a5ae-5a35d894726e","Type":"ContainerDied","Data":"0c380ac82e4677c5630bb9aed8688a4bb0c3ef53d917b56a260362706c342dec"}
Nov 02 09:50:05.843777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.843417    4449 scope.go:122] "RemoveContainer" containerID="11822a424089808f6099254a352c407abd819c54e1d747af6d0a87835ebd1ec0"
Nov 02 09:50:05.865497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.865475    4449 scope.go:122] "RemoveContainer" containerID="11822a424089808f6099254a352c407abd819c54e1d747af6d0a87835ebd1ec0"
Nov 02 09:50:05.865905 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:50:05.865879    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11822a424089808f6099254a352c407abd819c54e1d747af6d0a87835ebd1ec0\": container with ID starting with 11822a424089808f6099254a352c407abd819c54e1d747af6d0a87835ebd1ec0 not found: ID does not exist" containerID="11822a424089808f6099254a352c407abd819c54e1d747af6d0a87835ebd1ec0"
Nov 02 09:50:05.865981 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.865920    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11822a424089808f6099254a352c407abd819c54e1d747af6d0a87835ebd1ec0"} err="failed to get container status \"11822a424089808f6099254a352c407abd819c54e1d747af6d0a87835ebd1ec0\": rpc error: code = NotFound desc = could not find container \"11822a424089808f6099254a352c407abd819c54e1d747af6d0a87835ebd1ec0\": container with ID starting with 11822a424089808f6099254a352c407abd819c54e1d747af6d0a87835ebd1ec0 not found: ID does not exist"
Nov 02 09:50:05.873068 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.873042    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"]
Nov 02 09:50:05.878065 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:05.878037    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-301be8/test-model-301be8-kserve-5955c9fdfc-qfwkz"]
Nov 02 09:50:06.018388 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.018359    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"
Nov 02 09:50:06.079499 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.079459    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-xks-ca\") pod \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\" (UID: \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\") "
Nov 02 09:50:06.079657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.079519    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-openshift-ca\") pod \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\" (UID: \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\") "
Nov 02 09:50:06.079722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.079686    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-kube-api-access-mm667\" (UniqueName: \"kubernetes.io/projected/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-kube-api-access-mm667\") pod \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\" (UID: \"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257\") "
Nov 02 09:50:06.079965 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.079934    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-openshift-ca" pod "e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257" (UID: "e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:06.080112 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.080094    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:06.081815 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.081787    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-xks-ca" pod "e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257" (UID: "e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:50:06.081909 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.081880    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-kube-api-access-mm667" pod "e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257" (UID: "e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257"). InnerVolumeSpecName "kube-api-access-mm667". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:06.181386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.181356    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm667\" (UniqueName: \"kubernetes.io/projected/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-kube-api-access-mm667\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:06.181386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.181386    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:06.548038 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:50:06.548000    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:50:06.721812 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.721773    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"]
Nov 02 09:50:06.741556 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.741524    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"]
Nov 02 09:50:06.854730 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.854647    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn" event={"ID":"e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257","Type":"ContainerDied","Data":"4636f3fa268f9a1cc1e559f40bb7b0530ff503a10a9427586f5fb1474f6725b9"}
Nov 02 09:50:06.854730 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.854686    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"
Nov 02 09:50:06.854730 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:06.854696    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4636f3fa268f9a1cc1e559f40bb7b0530ff503a10a9427586f5fb1474f6725b9"
Nov 02 09:50:07.208771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:07.208687    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed165dd-1500-4fd4-a5ae-5a35d894726e" path="/var/lib/kubelet/pods/5ed165dd-1500-4fd4-a5ae-5a35d894726e/volumes"
Nov 02 09:50:09.775291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:09.775245    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:50:09.788119 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:09.788086    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:50:09.890582 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:09.890552    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:50:11.722438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:11.722395    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll" podUID="094a6ff2-c320-45e3-839a-9ba70ec6da87" containerName="payload-processing" containerID="cri-o://9064bddc66b23dfde817a3367c46c126568525071e56f8f23799b1632f6149d2" gracePeriod=25
Nov 02 09:50:11.742068 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:11.742026    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4" podUID="f1a365ff-43d5-400d-80d6-0cd798f2f7bf" containerName="payload-pre-processing" containerID="cri-o://62f07ce53eadb44da1b295ba773b1cc43769adde6b287a12278dee3f7030a3b4" gracePeriod=25
Nov 02 09:50:14.092543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.092498    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"]
Nov 02 09:50:14.093572 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.093551    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5ed165dd-1500-4fd4-a5ae-5a35d894726e" containerName="main"
Nov 02 09:50:14.093629 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.093575    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed165dd-1500-4fd4-a5ae-5a35d894726e" containerName="main"
Nov 02 09:50:14.093629 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.093601    4449 cpu_manager.go:414] "containerMap: removing container" podUID="e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257" containerName="revoke-keys"
Nov 02 09:50:14.093629 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.093609    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257" containerName="revoke-keys"
Nov 02 09:50:14.093827 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.093814    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed165dd-1500-4fd4-a5ae-5a35d894726e" containerName="main"
Nov 02 09:50:14.093858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.093835    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257" containerName="revoke-keys"
Nov 02 09:50:14.098251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.098198    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.114595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.114559    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"]
Nov 02 09:50:14.258848 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.258807    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.261040 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.261010    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-istio-token\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.261845 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.261824    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-certs\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.262244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.262204    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a361364-cc00-418a-9334-47bbce091b43-istiod-ca-cert\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.262466 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.262449    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-envoy\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.262645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.262616    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-data\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.262838 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.262820    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hwmn\" (UniqueName: \"kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-kube-api-access-7hwmn\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.263017 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.263002    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-credential-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.263192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.263176    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a361364-cc00-418a-9334-47bbce091b43-istio-podinfo\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.374332 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.374233    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-certs\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.374586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.374565    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a361364-cc00-418a-9334-47bbce091b43-istiod-ca-cert\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.374737 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.374722    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-envoy\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.374858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.374844    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-data\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.374997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.374981    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hwmn\" (UniqueName: \"kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-kube-api-access-7hwmn\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.375158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.375139    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-credential-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.375334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.375315    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a361364-cc00-418a-9334-47bbce091b43-istio-podinfo\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.375523 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.375504    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.388911 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.388884    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-istio-token\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.401237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.376313    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-certs\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.401237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.377005    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a361364-cc00-418a-9334-47bbce091b43-istiod-ca-cert\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.401237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.387735    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-credential-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.401237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.388816    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.401237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.376036    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-data\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.454916 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.453567    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-envoy\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.485242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.461988    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a361364-cc00-418a-9334-47bbce091b43-istio-podinfo\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.485242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.465791    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-istio-token\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.485242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.471281    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hwmn\" (UniqueName: \"kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-kube-api-access-7hwmn\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:14.721252 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:14.720432    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:16.045713 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:16.043624    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" podUID="e5827546-705c-4b91-a21e-daacfee4134a" containerName="main" containerID="cri-o://ade4a05ea793c98d98c9ac01e9d1e26da578a965d19c1d0af5e2655a00177126" gracePeriod=45
Nov 02 09:50:16.075689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:16.075343    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4" podUID="f1a365ff-43d5-400d-80d6-0cd798f2f7bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:50:16.077172 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:16.077094    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"]
Nov 02 09:50:16.191308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:16.182643    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:50:16.191308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:16.182745    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:50:16.204245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:16.202045    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:50:16.344349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:16.241537    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7" event={"ID":"6a361364-cc00-418a-9334-47bbce091b43","Type":"ContainerStarted","Data":"c16dc3c8122536cbd729c8cc502fecc369cefb11ea13b06665822ab773acb501"}
Nov 02 09:50:16.517599 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:16.289696    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll" podUID="094a6ff2-c320-45e3-839a-9ba70ec6da87" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:50:16.691245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:16.688937    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" podUID="e5827546-705c-4b91-a21e-daacfee4134a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": dial tcp 10.134.0.94:8000: connect: connection refused"
Nov 02 09:50:16.855265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:16.847570    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded"
Nov 02 09:50:16.917055 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:16.917002    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": context deadline exceeded"
Nov 02 09:50:16.947241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:16.923298    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:16.974552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:16.974506    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 02 09:50:17.307579 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.307536    4449 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:17.490243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.489751    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7" event={"ID":"6a361364-cc00-418a-9334-47bbce091b43","Type":"ContainerStarted","Data":"a2f5dd16320f157fa080672de59bc260e845237ba533fecc6b1ed3f92a27e4fe"}
Nov 02 09:50:17.572253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.572184    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" podUID="e5827546-705c-4b91-a21e-daacfee4134a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": dial tcp 10.134.0.94:8000: connect: connection refused"
Nov 02 09:50:17.572454 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.572331    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:50:17.608268 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.601902    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7" podStartSLOduration=3.6018781349999998 podStartE2EDuration="3.601878135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:17.55059095 +0000 UTC m=+894.488877771" watchObservedRunningTime="2026-11-02 09:50:17.601878135 +0000 UTC m=+894.540164911"
Nov 02 09:50:17.738303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.733759    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:17.910280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.892341    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:17.910280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.893097    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:50:17.923263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.920339    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": context deadline exceeded"
Nov 02 09:50:17.923263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.921097    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:50:17.947516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.928954    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:17.947516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.929315    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:50:18.132768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.963139    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"]
Nov 02 09:50:18.132768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.964340    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:50:18.132768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:17.964395    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" podUID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:18.132768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.013645    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:18.132768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.014049    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:50:18.132768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.015113    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk" podUID="7ef35a63-382b-4978-a07d-197c7775de78" containerName="istio-proxy" containerID="cri-o://16448d7ef350e8c89b0630a4528613835d07d4bc575d923a09e20f229c93e107" gracePeriod=30
Nov 02 09:50:18.132768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.057843    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:18.228688 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.228659    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:50:18.572240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.567617    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" podUID="e5827546-705c-4b91-a21e-daacfee4134a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": dial tcp 10.134.0.94:8000: connect: connection refused"
Nov 02 09:50:18.572990 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.572961    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"]
Nov 02 09:50:18.573157 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.573076    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:50:18.600777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.600740    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:50:18.621023 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.620666    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:50:18.675196 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.675159    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"]
Nov 02 09:50:18.723936 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.723857    4449 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 02 09:50:18.723936 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.723909    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7" podUID="6a361364-cc00-418a-9334-47bbce091b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 02 09:50:18.731029 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.729926    4449 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7" secret="" err="secret \"e2e-inf-a-301be8-openshift-default-dockercfg-fxzjg\" not found"
Nov 02 09:50:18.997190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:18.997165    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:50:19.082984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.082951    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5827546-705c-4b91-a21e-daacfee4134a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5827546-705c-4b91-a21e-daacfee4134a-tls-certs\") pod \"e5827546-705c-4b91-a21e-daacfee4134a\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") "
Nov 02 09:50:19.083136 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.083001    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-tmp-dir\") pod \"e5827546-705c-4b91-a21e-daacfee4134a\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") "
Nov 02 09:50:19.083136 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.083083    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-dshm\") pod \"e5827546-705c-4b91-a21e-daacfee4134a\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") "
Nov 02 09:50:19.083136 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.083110    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5827546-705c-4b91-a21e-daacfee4134a-kube-api-access-z26mz\" (UniqueName: \"kubernetes.io/projected/e5827546-705c-4b91-a21e-daacfee4134a-kube-api-access-z26mz\") pod \"e5827546-705c-4b91-a21e-daacfee4134a\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") "
Nov 02 09:50:19.083338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.083156    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-home\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-home\") pod \"e5827546-705c-4b91-a21e-daacfee4134a\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") "
Nov 02 09:50:19.083338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.083179    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-model-cache\") pod \"e5827546-705c-4b91-a21e-daacfee4134a\" (UID: \"e5827546-705c-4b91-a21e-daacfee4134a\") "
Nov 02 09:50:19.083453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.083397    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-tmp-dir" pod "e5827546-705c-4b91-a21e-daacfee4134a" (UID: "e5827546-705c-4b91-a21e-daacfee4134a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:19.083453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.083410    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-home" pod "e5827546-705c-4b91-a21e-daacfee4134a" (UID: "e5827546-705c-4b91-a21e-daacfee4134a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:19.083728 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.083701    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-model-cache" pod "e5827546-705c-4b91-a21e-daacfee4134a" (UID: "e5827546-705c-4b91-a21e-daacfee4134a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:19.084168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.084145    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:19.084246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.084176    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:19.084246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.084192    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:19.085612 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.085581    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5827546-705c-4b91-a21e-daacfee4134a-kube-api-access-z26mz" pod "e5827546-705c-4b91-a21e-daacfee4134a" (UID: "e5827546-705c-4b91-a21e-daacfee4134a"). InnerVolumeSpecName "kube-api-access-z26mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:19.085952 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.085931    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5827546-705c-4b91-a21e-daacfee4134a-tls-certs" pod "e5827546-705c-4b91-a21e-daacfee4134a" (UID: "e5827546-705c-4b91-a21e-daacfee4134a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:50:19.085952 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.085947    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-dshm" pod "e5827546-705c-4b91-a21e-daacfee4134a" (UID: "e5827546-705c-4b91-a21e-daacfee4134a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:19.185757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.185726    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5827546-705c-4b91-a21e-daacfee4134a-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:19.185757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.185753    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5827546-705c-4b91-a21e-daacfee4134a-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:19.185757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.185763    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z26mz\" (UniqueName: \"kubernetes.io/projected/e5827546-705c-4b91-a21e-daacfee4134a-kube-api-access-z26mz\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:19.721378 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.721342    4449 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 02 09:50:19.721875 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.721406    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7" podUID="6a361364-cc00-418a-9334-47bbce091b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 02 09:50:19.738243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.738180    4449 generic.go:356] "Generic (PLEG): container finished" podID="e5827546-705c-4b91-a21e-daacfee4134a" containerID="ade4a05ea793c98d98c9ac01e9d1e26da578a965d19c1d0af5e2655a00177126" exitCode=0
Nov 02 09:50:19.738408 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.738255    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"
Nov 02 09:50:19.738408 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.738266    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" event={"ID":"e5827546-705c-4b91-a21e-daacfee4134a","Type":"ContainerDied","Data":"ade4a05ea793c98d98c9ac01e9d1e26da578a965d19c1d0af5e2655a00177126"}
Nov 02 09:50:19.738408 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.738317    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp" event={"ID":"e5827546-705c-4b91-a21e-daacfee4134a","Type":"ContainerDied","Data":"9c84083ade0178ff9fc3b4c6752332ba880490ddae74214ddd0facf6857f59b6"}
Nov 02 09:50:19.738408 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.738336    4449 scope.go:122] "RemoveContainer" containerID="ade4a05ea793c98d98c9ac01e9d1e26da578a965d19c1d0af5e2655a00177126"
Nov 02 09:50:19.738602 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.738580    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7" podUID="6a361364-cc00-418a-9334-47bbce091b43" containerName="istio-proxy" containerID="cri-o://a2f5dd16320f157fa080672de59bc260e845237ba533fecc6b1ed3f92a27e4fe" gracePeriod=30
Nov 02 09:50:19.755999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.755766    4449 scope.go:122] "RemoveContainer" containerID="ade4a05ea793c98d98c9ac01e9d1e26da578a965d19c1d0af5e2655a00177126"
Nov 02 09:50:19.756919 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:50:19.756873    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade4a05ea793c98d98c9ac01e9d1e26da578a965d19c1d0af5e2655a00177126\": container with ID starting with ade4a05ea793c98d98c9ac01e9d1e26da578a965d19c1d0af5e2655a00177126 not found: ID does not exist" containerID="ade4a05ea793c98d98c9ac01e9d1e26da578a965d19c1d0af5e2655a00177126"
Nov 02 09:50:19.757059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.756914    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade4a05ea793c98d98c9ac01e9d1e26da578a965d19c1d0af5e2655a00177126"} err="failed to get container status \"ade4a05ea793c98d98c9ac01e9d1e26da578a965d19c1d0af5e2655a00177126\": rpc error: code = NotFound desc = could not find container \"ade4a05ea793c98d98c9ac01e9d1e26da578a965d19c1d0af5e2655a00177126\": container with ID starting with ade4a05ea793c98d98c9ac01e9d1e26da578a965d19c1d0af5e2655a00177126 not found: ID does not exist"
Nov 02 09:50:19.764116 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.764090    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"]
Nov 02 09:50:19.770893 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:19.770859    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8743a5/test-model-8743a5-kserve-59f899b667-rznvp"]
Nov 02 09:50:20.372586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.372544    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"]
Nov 02 09:50:20.373249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.373231    4449 cpu_manager.go:414] "containerMap: removing container" podUID="e5827546-705c-4b91-a21e-daacfee4134a" containerName="main"
Nov 02 09:50:20.373249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.373246    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5827546-705c-4b91-a21e-daacfee4134a" containerName="main"
Nov 02 09:50:20.373444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.373364    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5827546-705c-4b91-a21e-daacfee4134a" containerName="main"
Nov 02 09:50:20.378569 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.378546    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"
Nov 02 09:50:20.382392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.382353    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:50:20.383375 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.383354    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:50:20.386152 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.386124    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"]
Nov 02 09:50:20.396967 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.396918    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:50:20.397272 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.397252    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:50:20.406328 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.406302    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fe06209-72gd6\" (UID: \"79021ba5-953f-461e-b75e-957242a29b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"
Nov 02 09:50:20.406587 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.406571    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwtbp\" (UniqueName: \"kubernetes.io/projected/79021ba5-953f-461e-b75e-957242a29b3a-kube-api-access-vwtbp\") pod \"maas-api-revoke-sub-6fe06209-72gd6\" (UID: \"79021ba5-953f-461e-b75e-957242a29b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"
Nov 02 09:50:20.406634 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.406602    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fe06209-72gd6\" (UID: \"79021ba5-953f-461e-b75e-957242a29b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"
Nov 02 09:50:20.507373 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.507345    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwtbp\" (UniqueName: \"kubernetes.io/projected/79021ba5-953f-461e-b75e-957242a29b3a-kube-api-access-vwtbp\") pod \"maas-api-revoke-sub-6fe06209-72gd6\" (UID: \"79021ba5-953f-461e-b75e-957242a29b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"
Nov 02 09:50:20.507523 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.507382    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fe06209-72gd6\" (UID: \"79021ba5-953f-461e-b75e-957242a29b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"
Nov 02 09:50:20.507523 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.507431    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fe06209-72gd6\" (UID: \"79021ba5-953f-461e-b75e-957242a29b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"
Nov 02 09:50:20.508173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.508146    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fe06209-72gd6\" (UID: \"79021ba5-953f-461e-b75e-957242a29b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"
Nov 02 09:50:20.509995 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.509965    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fe06209-72gd6\" (UID: \"79021ba5-953f-461e-b75e-957242a29b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"
Nov 02 09:50:20.521861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.521835    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwtbp\" (UniqueName: \"kubernetes.io/projected/79021ba5-953f-461e-b75e-957242a29b3a-kube-api-access-vwtbp\") pod \"maas-api-revoke-sub-6fe06209-72gd6\" (UID: \"79021ba5-953f-461e-b75e-957242a29b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"
Nov 02 09:50:20.697032 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.696946    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"
Nov 02 09:50:20.986051 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:20.986020    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79021ba5_953f_461e_b75e_957242a29b3a.slice/crio-30d8913063a76e11df193c034c04c9912966a249e5bdce98ec6207e59731f46d WatchSource:0}: Error finding container 30d8913063a76e11df193c034c04c9912966a249e5bdce98ec6207e59731f46d: Status 404 returned error can't find the container with id 30d8913063a76e11df193c034c04c9912966a249e5bdce98ec6207e59731f46d
Nov 02 09:50:20.987538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:20.987513    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"]
Nov 02 09:50:21.207808 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:21.207774    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5827546-705c-4b91-a21e-daacfee4134a" path="/var/lib/kubelet/pods/e5827546-705c-4b91-a21e-daacfee4134a/volumes"
Nov 02 09:50:21.760662 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:21.760617    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6" event={"ID":"79021ba5-953f-461e-b75e-957242a29b3a","Type":"ContainerStarted","Data":"2e9732213da9391c2688bf06e273874581154966e46ae1c2c4fda70911661b90"}
Nov 02 09:50:21.760841 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:21.760686    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6" event={"ID":"79021ba5-953f-461e-b75e-957242a29b3a","Type":"ContainerStarted","Data":"30d8913063a76e11df193c034c04c9912966a249e5bdce98ec6207e59731f46d"}
Nov 02 09:50:21.785445 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:21.785382    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6" podStartSLOduration=1.7853617179999999 podStartE2EDuration="1.785361718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:21.77785171 +0000 UTC m=+898.716138509" watchObservedRunningTime="2026-11-02 09:50:21.785361718 +0000 UTC m=+898.723648495"
Nov 02 09:50:22.460796 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:22.460760    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"]
Nov 02 09:50:22.461327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:22.460984    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz" podUID="60302c85-67c9-40da-b62c-1ff3ffdbcf1f" containerName="istio-proxy" containerID="cri-o://ebc820c17c8afbe794d691337d245e870fcef4ff9c0ba9a18e6469b33aca93db" gracePeriod=30
Nov 02 09:50:22.467982 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:22.467948    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"]
Nov 02 09:50:22.468198 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:22.468173    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" podUID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" containerName="istio-proxy" containerID="cri-o://f12027ca32095dbe15199fb3c209cfb642e4e2251ad89e9011172ea8b10ee92d" gracePeriod=30
Nov 02 09:50:22.775579 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:22.775487    4449 generic.go:356] "Generic (PLEG): container finished" podID="79021ba5-953f-461e-b75e-957242a29b3a" containerID="2e9732213da9391c2688bf06e273874581154966e46ae1c2c4fda70911661b90" exitCode=0
Nov 02 09:50:22.775728 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:22.775571    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6" event={"ID":"79021ba5-953f-461e-b75e-957242a29b3a","Type":"ContainerDied","Data":"2e9732213da9391c2688bf06e273874581154966e46ae1c2c4fda70911661b90"}
Nov 02 09:50:23.297316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.297282    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:50:23.351486 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.351395    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-data\") pod \"7ef35a63-382b-4978-a07d-197c7775de78\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") "
Nov 02 09:50:23.351782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.351758    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-kube-api-access-7cxh4\" (UniqueName: \"kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-kube-api-access-7cxh4\") pod \"7ef35a63-382b-4978-a07d-197c7775de78\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") "
Nov 02 09:50:23.351899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.351822    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-envoy\") pod \"7ef35a63-382b-4978-a07d-197c7775de78\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") "
Nov 02 09:50:23.351899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.351890    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-certs\") pod \"7ef35a63-382b-4978-a07d-197c7775de78\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") "
Nov 02 09:50:23.352004 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.351948    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-credential-socket\") pod \"7ef35a63-382b-4978-a07d-197c7775de78\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") "
Nov 02 09:50:23.352054 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.352040    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-istio-token\" (UniqueName: \"kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-istio-token\") pod \"7ef35a63-382b-4978-a07d-197c7775de78\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") "
Nov 02 09:50:23.352129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.352113    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ef35a63-382b-4978-a07d-197c7775de78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef35a63-382b-4978-a07d-197c7775de78-istiod-ca-cert\") pod \"7ef35a63-382b-4978-a07d-197c7775de78\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") "
Nov 02 09:50:23.352257 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.352206    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-socket\") pod \"7ef35a63-382b-4978-a07d-197c7775de78\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") "
Nov 02 09:50:23.352337 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.352271    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ef35a63-382b-4978-a07d-197c7775de78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef35a63-382b-4978-a07d-197c7775de78-istio-podinfo\") pod \"7ef35a63-382b-4978-a07d-197c7775de78\" (UID: \"7ef35a63-382b-4978-a07d-197c7775de78\") "
Nov 02 09:50:23.355527 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.355484    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-credential-socket" pod "7ef35a63-382b-4978-a07d-197c7775de78" (UID: "7ef35a63-382b-4978-a07d-197c7775de78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:23.355948 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.355807    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-socket" pod "7ef35a63-382b-4978-a07d-197c7775de78" (UID: "7ef35a63-382b-4978-a07d-197c7775de78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:23.356269 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.355984    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-data" pod "7ef35a63-382b-4978-a07d-197c7775de78" (UID: "7ef35a63-382b-4978-a07d-197c7775de78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:23.358646 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.357738    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-istio-token" pod "7ef35a63-382b-4978-a07d-197c7775de78" (UID: "7ef35a63-382b-4978-a07d-197c7775de78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:23.365880 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.365834    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef35a63-382b-4978-a07d-197c7775de78-istiod-ca-cert" pod "7ef35a63-382b-4978-a07d-197c7775de78" (UID: "7ef35a63-382b-4978-a07d-197c7775de78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:23.366311 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.366273    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ef35a63-382b-4978-a07d-197c7775de78-istio-podinfo" pod "7ef35a63-382b-4978-a07d-197c7775de78" (UID: "7ef35a63-382b-4978-a07d-197c7775de78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:50:23.369558 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.369530    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-envoy" pod "7ef35a63-382b-4978-a07d-197c7775de78" (UID: "7ef35a63-382b-4978-a07d-197c7775de78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:23.369677 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.369572    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-certs" pod "7ef35a63-382b-4978-a07d-197c7775de78" (UID: "7ef35a63-382b-4978-a07d-197c7775de78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:23.369677 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.369574    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-kube-api-access-7cxh4" pod "7ef35a63-382b-4978-a07d-197c7775de78" (UID: "7ef35a63-382b-4978-a07d-197c7775de78"). InnerVolumeSpecName "kube-api-access-7cxh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:23.454039 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.454003    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.454039 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.454039    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.454291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.454048    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.454291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.454058    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ef35a63-382b-4978-a07d-197c7775de78-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.454291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.454067    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.454291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.454075    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ef35a63-382b-4978-a07d-197c7775de78-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.454291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.454086    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.454291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.454095    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cxh4\" (UniqueName: \"kubernetes.io/projected/7ef35a63-382b-4978-a07d-197c7775de78-kube-api-access-7cxh4\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.454291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.454106    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ef35a63-382b-4978-a07d-197c7775de78-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.676178 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.676071    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq" podUID="0c609200-d239-4c5a-8743-acc3d200579f" containerName="payload-processing" containerID="cri-o://bad54496c3f320c98c1c3c463c93e0f17fae7fca5539d74ba08150f7dcbb3034" gracePeriod=25
Nov 02 09:50:23.786936 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.786902    4449 generic.go:356] "Generic (PLEG): container finished" podID="7ef35a63-382b-4978-a07d-197c7775de78" containerID="16448d7ef350e8c89b0630a4528613835d07d4bc575d923a09e20f229c93e107" exitCode=0
Nov 02 09:50:23.787099 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.786948    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk" event={"ID":"7ef35a63-382b-4978-a07d-197c7775de78","Type":"ContainerDied","Data":"16448d7ef350e8c89b0630a4528613835d07d4bc575d923a09e20f229c93e107"}
Nov 02 09:50:23.787099 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.786997    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"
Nov 02 09:50:23.787099 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.787044    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk" event={"ID":"7ef35a63-382b-4978-a07d-197c7775de78","Type":"ContainerDied","Data":"3d3c1266eead443d15c5e7240d551a161779208a5b5f6d6e9442af1d36a62675"}
Nov 02 09:50:23.787099 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.787070    4449 scope.go:122] "RemoveContainer" containerID="16448d7ef350e8c89b0630a4528613835d07d4bc575d923a09e20f229c93e107"
Nov 02 09:50:23.802426 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.802391    4449 scope.go:122] "RemoveContainer" containerID="6b64cb0d4613b6a507e71a031ad6fb3647d5a21be29a06a19746e905fe16e23f"
Nov 02 09:50:23.822055 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.822000    4449 scope.go:122] "RemoveContainer" containerID="16448d7ef350e8c89b0630a4528613835d07d4bc575d923a09e20f229c93e107"
Nov 02 09:50:23.822286 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:50:23.822260    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16448d7ef350e8c89b0630a4528613835d07d4bc575d923a09e20f229c93e107\": container with ID starting with 16448d7ef350e8c89b0630a4528613835d07d4bc575d923a09e20f229c93e107 not found: ID does not exist" containerID="16448d7ef350e8c89b0630a4528613835d07d4bc575d923a09e20f229c93e107"
Nov 02 09:50:23.822364 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.822300    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16448d7ef350e8c89b0630a4528613835d07d4bc575d923a09e20f229c93e107"} err="failed to get container status \"16448d7ef350e8c89b0630a4528613835d07d4bc575d923a09e20f229c93e107\": rpc error: code = NotFound desc = could not find container \"16448d7ef350e8c89b0630a4528613835d07d4bc575d923a09e20f229c93e107\": container with ID starting with 16448d7ef350e8c89b0630a4528613835d07d4bc575d923a09e20f229c93e107 not found: ID does not exist"
Nov 02 09:50:23.822364 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.822326    4449 scope.go:122] "RemoveContainer" containerID="6b64cb0d4613b6a507e71a031ad6fb3647d5a21be29a06a19746e905fe16e23f"
Nov 02 09:50:23.822552 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:50:23.822531    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b64cb0d4613b6a507e71a031ad6fb3647d5a21be29a06a19746e905fe16e23f\": container with ID starting with 6b64cb0d4613b6a507e71a031ad6fb3647d5a21be29a06a19746e905fe16e23f not found: ID does not exist" containerID="6b64cb0d4613b6a507e71a031ad6fb3647d5a21be29a06a19746e905fe16e23f"
Nov 02 09:50:23.822613 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.822559    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b64cb0d4613b6a507e71a031ad6fb3647d5a21be29a06a19746e905fe16e23f"} err="failed to get container status \"6b64cb0d4613b6a507e71a031ad6fb3647d5a21be29a06a19746e905fe16e23f\": rpc error: code = NotFound desc = could not find container \"6b64cb0d4613b6a507e71a031ad6fb3647d5a21be29a06a19746e905fe16e23f\": container with ID starting with 6b64cb0d4613b6a507e71a031ad6fb3647d5a21be29a06a19746e905fe16e23f not found: ID does not exist"
Nov 02 09:50:23.831869 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.831836    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"]
Nov 02 09:50:23.840409 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.840371    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-k5dvk"]
Nov 02 09:50:23.979975 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.979939    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"
Nov 02 09:50:23.984497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:23.984466    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:50:24.062124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.062095    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-provider-credentials\") pod \"0c609200-d239-4c5a-8743-acc3d200579f\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") "
Nov 02 09:50:24.062313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.062296    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-kube-api-access-xxmn7\" (UniqueName: \"kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-kube-api-access-xxmn7\") pod \"0c609200-d239-4c5a-8743-acc3d200579f\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") "
Nov 02 09:50:24.062387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.062338    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-xks-ca\") pod \"79021ba5-953f-461e-b75e-957242a29b3a\" (UID: \"79021ba5-953f-461e-b75e-957242a29b3a\") "
Nov 02 09:50:24.062387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.062367    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-routing-overlay-volume\") pod \"0c609200-d239-4c5a-8743-acc3d200579f\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") "
Nov 02 09:50:24.062467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.062385    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-openshift-ca\") pod \"79021ba5-953f-461e-b75e-957242a29b3a\" (UID: \"79021ba5-953f-461e-b75e-957242a29b3a\") "
Nov 02 09:50:24.062467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.062454    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-plugins-config-volume\") pod \"0c609200-d239-4c5a-8743-acc3d200579f\" (UID: \"0c609200-d239-4c5a-8743-acc3d200579f\") "
Nov 02 09:50:24.062577 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.062549    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79021ba5-953f-461e-b75e-957242a29b3a-kube-api-access-vwtbp\" (UniqueName: \"kubernetes.io/projected/79021ba5-953f-461e-b75e-957242a29b3a-kube-api-access-vwtbp\") pod \"79021ba5-953f-461e-b75e-957242a29b3a\" (UID: \"79021ba5-953f-461e-b75e-957242a29b3a\") "
Nov 02 09:50:24.062828 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.062796    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-openshift-ca" pod "79021ba5-953f-461e-b75e-957242a29b3a" (UID: "79021ba5-953f-461e-b75e-957242a29b3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:24.062828 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.062811    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-routing-overlay-volume" pod "0c609200-d239-4c5a-8743-acc3d200579f" (UID: "0c609200-d239-4c5a-8743-acc3d200579f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:24.062965 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.062917    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-plugins-config-volume" pod "0c609200-d239-4c5a-8743-acc3d200579f" (UID: "0c609200-d239-4c5a-8743-acc3d200579f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:24.063515 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.063485    4449 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-routing-overlay-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:24.063656 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.063519    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:24.063656 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.063536    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c609200-d239-4c5a-8743-acc3d200579f-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:24.064684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.064661    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-xks-ca" pod "79021ba5-953f-461e-b75e-957242a29b3a" (UID: "79021ba5-953f-461e-b75e-957242a29b3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:50:24.064848 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.064826    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-kube-api-access-xxmn7" pod "0c609200-d239-4c5a-8743-acc3d200579f" (UID: "0c609200-d239-4c5a-8743-acc3d200579f"). InnerVolumeSpecName "kube-api-access-xxmn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:24.064848 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.064837    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-provider-credentials" pod "0c609200-d239-4c5a-8743-acc3d200579f" (UID: "0c609200-d239-4c5a-8743-acc3d200579f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:24.064931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.064907    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79021ba5-953f-461e-b75e-957242a29b3a-kube-api-access-vwtbp" pod "79021ba5-953f-461e-b75e-957242a29b3a" (UID: "79021ba5-953f-461e-b75e-957242a29b3a"). InnerVolumeSpecName "kube-api-access-vwtbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:24.164610 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.164579    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxmn7\" (UniqueName: \"kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-kube-api-access-xxmn7\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:24.164610 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.164607    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79021ba5-953f-461e-b75e-957242a29b3a-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:24.164610 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.164618    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwtbp\" (UniqueName: \"kubernetes.io/projected/79021ba5-953f-461e-b75e-957242a29b3a-kube-api-access-vwtbp\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:24.164836 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.164627    4449 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c609200-d239-4c5a-8743-acc3d200579f-provider-credentials\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:24.797932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.797896    4449 generic.go:356] "Generic (PLEG): container finished" podID="0c609200-d239-4c5a-8743-acc3d200579f" containerID="bad54496c3f320c98c1c3c463c93e0f17fae7fca5539d74ba08150f7dcbb3034" exitCode=0
Nov 02 09:50:24.798411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.797962    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"
Nov 02 09:50:24.798411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.797993    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq" event={"ID":"0c609200-d239-4c5a-8743-acc3d200579f","Type":"ContainerDied","Data":"bad54496c3f320c98c1c3c463c93e0f17fae7fca5539d74ba08150f7dcbb3034"}
Nov 02 09:50:24.798411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.798057    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq" event={"ID":"0c609200-d239-4c5a-8743-acc3d200579f","Type":"ContainerDied","Data":"86c91d5ada0ca29df4905e2dce1e50d558afaab0579d54c400e2ffab6b6a7d84"}
Nov 02 09:50:24.798411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.798093    4449 scope.go:122] "RemoveContainer" containerID="bad54496c3f320c98c1c3c463c93e0f17fae7fca5539d74ba08150f7dcbb3034"
Nov 02 09:50:24.800458 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.800431    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6" event={"ID":"79021ba5-953f-461e-b75e-957242a29b3a","Type":"ContainerDied","Data":"30d8913063a76e11df193c034c04c9912966a249e5bdce98ec6207e59731f46d"}
Nov 02 09:50:24.800618 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.800481    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30d8913063a76e11df193c034c04c9912966a249e5bdce98ec6207e59731f46d"
Nov 02 09:50:24.800618 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.800553    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"
Nov 02 09:50:24.839158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.839127    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"]
Nov 02 09:50:24.845793 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.845771    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899fjwbq"]
Nov 02 09:50:24.859158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.859138    4449 scope.go:122] "RemoveContainer" containerID="bad54496c3f320c98c1c3c463c93e0f17fae7fca5539d74ba08150f7dcbb3034"
Nov 02 09:50:24.859543 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:50:24.859520    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad54496c3f320c98c1c3c463c93e0f17fae7fca5539d74ba08150f7dcbb3034\": container with ID starting with bad54496c3f320c98c1c3c463c93e0f17fae7fca5539d74ba08150f7dcbb3034 not found: ID does not exist" containerID="bad54496c3f320c98c1c3c463c93e0f17fae7fca5539d74ba08150f7dcbb3034"
Nov 02 09:50:24.859640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.859553    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad54496c3f320c98c1c3c463c93e0f17fae7fca5539d74ba08150f7dcbb3034"} err="failed to get container status \"bad54496c3f320c98c1c3c463c93e0f17fae7fca5539d74ba08150f7dcbb3034\": rpc error: code = NotFound desc = could not find container \"bad54496c3f320c98c1c3c463c93e0f17fae7fca5539d74ba08150f7dcbb3034\": container with ID starting with bad54496c3f320c98c1c3c463c93e0f17fae7fca5539d74ba08150f7dcbb3034 not found: ID does not exist"
Nov 02 09:50:24.860618 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:24.860593    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"]
Nov 02 09:50:25.051888 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.051861    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:25.177594 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.177560    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a361364-cc00-418a-9334-47bbce091b43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a361364-cc00-418a-9334-47bbce091b43-istio-podinfo\") pod \"6a361364-cc00-418a-9334-47bbce091b43\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") "
Nov 02 09:50:25.177786 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.177619    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-kube-api-access-7hwmn\" (UniqueName: \"kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-kube-api-access-7hwmn\") pod \"6a361364-cc00-418a-9334-47bbce091b43\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") "
Nov 02 09:50:25.177786 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.177728    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-data\") pod \"6a361364-cc00-418a-9334-47bbce091b43\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") "
Nov 02 09:50:25.177908 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.177865    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-envoy\") pod \"6a361364-cc00-418a-9334-47bbce091b43\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") "
Nov 02 09:50:25.177960 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.177933    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-socket\") pod \"6a361364-cc00-418a-9334-47bbce091b43\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") "
Nov 02 09:50:25.178016 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.177976    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-istio-token\" (UniqueName: \"kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-istio-token\") pod \"6a361364-cc00-418a-9334-47bbce091b43\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") "
Nov 02 09:50:25.178113 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.178080    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-credential-socket\") pod \"6a361364-cc00-418a-9334-47bbce091b43\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") "
Nov 02 09:50:25.178243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.177974    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-data" pod "6a361364-cc00-418a-9334-47bbce091b43" (UID: "6a361364-cc00-418a-9334-47bbce091b43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:25.178330 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.178161    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-socket" pod "6a361364-cc00-418a-9334-47bbce091b43" (UID: "6a361364-cc00-418a-9334-47bbce091b43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:25.178330 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.178300    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a361364-cc00-418a-9334-47bbce091b43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a361364-cc00-418a-9334-47bbce091b43-istiod-ca-cert\") pod \"6a361364-cc00-418a-9334-47bbce091b43\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") "
Nov 02 09:50:25.178330 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.178314    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-credential-socket" pod "6a361364-cc00-418a-9334-47bbce091b43" (UID: "6a361364-cc00-418a-9334-47bbce091b43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:25.178670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.178336    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-certs\") pod \"6a361364-cc00-418a-9334-47bbce091b43\" (UID: \"6a361364-cc00-418a-9334-47bbce091b43\") "
Nov 02 09:50:25.178802 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.178642    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-certs" pod "6a361364-cc00-418a-9334-47bbce091b43" (UID: "6a361364-cc00-418a-9334-47bbce091b43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:25.179066 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.178937    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a361364-cc00-418a-9334-47bbce091b43-istiod-ca-cert" pod "6a361364-cc00-418a-9334-47bbce091b43" (UID: "6a361364-cc00-418a-9334-47bbce091b43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:25.179393 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.179371    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:25.179509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.179398    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a361364-cc00-418a-9334-47bbce091b43-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:25.179509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.179414    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:25.179509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.179427    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:25.179509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.179442    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:25.180264 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.180208    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a361364-cc00-418a-9334-47bbce091b43-istio-podinfo" pod "6a361364-cc00-418a-9334-47bbce091b43" (UID: "6a361364-cc00-418a-9334-47bbce091b43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:50:25.180264 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.180246    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-kube-api-access-7hwmn" pod "6a361364-cc00-418a-9334-47bbce091b43" (UID: "6a361364-cc00-418a-9334-47bbce091b43"). InnerVolumeSpecName "kube-api-access-7hwmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:25.180404 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.180283    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-envoy" pod "6a361364-cc00-418a-9334-47bbce091b43" (UID: "6a361364-cc00-418a-9334-47bbce091b43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:25.180505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.180491    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-istio-token" pod "6a361364-cc00-418a-9334-47bbce091b43" (UID: "6a361364-cc00-418a-9334-47bbce091b43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:25.211755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.211715    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c609200-d239-4c5a-8743-acc3d200579f" path="/var/lib/kubelet/pods/0c609200-d239-4c5a-8743-acc3d200579f/volumes"
Nov 02 09:50:25.212533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.212484    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef35a63-382b-4978-a07d-197c7775de78" path="/var/lib/kubelet/pods/7ef35a63-382b-4978-a07d-197c7775de78/volumes"
Nov 02 09:50:25.283024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.282869    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:25.283024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.282973    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a361364-cc00-418a-9334-47bbce091b43-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:25.283024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.282991    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hwmn\" (UniqueName: \"kubernetes.io/projected/6a361364-cc00-418a-9334-47bbce091b43-kube-api-access-7hwmn\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:25.283024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.283006    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a361364-cc00-418a-9334-47bbce091b43-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:25.812478 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.812443    4449 generic.go:356] "Generic (PLEG): container finished" podID="6a361364-cc00-418a-9334-47bbce091b43" containerID="a2f5dd16320f157fa080672de59bc260e845237ba533fecc6b1ed3f92a27e4fe" exitCode=0
Nov 02 09:50:25.812887 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.812500    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"
Nov 02 09:50:25.812887 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.812565    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7" event={"ID":"6a361364-cc00-418a-9334-47bbce091b43","Type":"ContainerDied","Data":"a2f5dd16320f157fa080672de59bc260e845237ba533fecc6b1ed3f92a27e4fe"}
Nov 02 09:50:25.812887 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.812620    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7" event={"ID":"6a361364-cc00-418a-9334-47bbce091b43","Type":"ContainerDied","Data":"c16dc3c8122536cbd729c8cc502fecc369cefb11ea13b06665822ab773acb501"}
Nov 02 09:50:25.812887 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.812644    4449 scope.go:122] "RemoveContainer" containerID="a2f5dd16320f157fa080672de59bc260e845237ba533fecc6b1ed3f92a27e4fe"
Nov 02 09:50:25.824765 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.824734    4449 scope.go:122] "RemoveContainer" containerID="a2f5dd16320f157fa080672de59bc260e845237ba533fecc6b1ed3f92a27e4fe"
Nov 02 09:50:25.825036 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:50:25.825016    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f5dd16320f157fa080672de59bc260e845237ba533fecc6b1ed3f92a27e4fe\": container with ID starting with a2f5dd16320f157fa080672de59bc260e845237ba533fecc6b1ed3f92a27e4fe not found: ID does not exist" containerID="a2f5dd16320f157fa080672de59bc260e845237ba533fecc6b1ed3f92a27e4fe"
Nov 02 09:50:25.825107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.825049    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f5dd16320f157fa080672de59bc260e845237ba533fecc6b1ed3f92a27e4fe"} err="failed to get container status \"a2f5dd16320f157fa080672de59bc260e845237ba533fecc6b1ed3f92a27e4fe\": rpc error: code = NotFound desc = could not find container \"a2f5dd16320f157fa080672de59bc260e845237ba533fecc6b1ed3f92a27e4fe\": container with ID starting with a2f5dd16320f157fa080672de59bc260e845237ba533fecc6b1ed3f92a27e4fe not found: ID does not exist"
Nov 02 09:50:25.840757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.840725    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"]
Nov 02 09:50:25.854813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.854787    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-rmcf7"]
Nov 02 09:50:25.894499 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.894465    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4" podUID="f1a365ff-43d5-400d-80d6-0cd798f2f7bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:50:25.894675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:25.894466    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll" podUID="094a6ff2-c320-45e3-839a-9ba70ec6da87" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:50:27.235243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:27.234869    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a361364-cc00-418a-9334-47bbce091b43" path="/var/lib/kubelet/pods/6a361364-cc00-418a-9334-47bbce091b43/volumes"
Nov 02 09:50:27.963393 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:27.963356    4449 generic.go:356] "Generic (PLEG): container finished" podID="60302c85-67c9-40da-b62c-1ff3ffdbcf1f" containerID="ebc820c17c8afbe794d691337d245e870fcef4ff9c0ba9a18e6469b33aca93db" exitCode=0
Nov 02 09:50:27.963837 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:27.963808    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz" event={"ID":"60302c85-67c9-40da-b62c-1ff3ffdbcf1f","Type":"ContainerDied","Data":"ebc820c17c8afbe794d691337d245e870fcef4ff9c0ba9a18e6469b33aca93db"}
Nov 02 09:50:27.966307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:27.966280    4449 generic.go:356] "Generic (PLEG): container finished" podID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" containerID="f12027ca32095dbe15199fb3c209cfb642e4e2251ad89e9011172ea8b10ee92d" exitCode=0
Nov 02 09:50:27.966630 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:27.966610    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" event={"ID":"dd0a870c-883a-4314-9e4f-d05fb3241d5a","Type":"ContainerDied","Data":"f12027ca32095dbe15199fb3c209cfb642e4e2251ad89e9011172ea8b10ee92d"}
Nov 02 09:50:28.705541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:28.705440    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded"
Nov 02 09:50:28.763671 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:28.762471    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:28.811102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:28.809336    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:29.055704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.052875    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:29.239480 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.239376    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:29.401978 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.401120    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz" event={"ID":"60302c85-67c9-40da-b62c-1ff3ffdbcf1f","Type":"ContainerDied","Data":"25e7beea7effd4df14500c2aeae7aa34cbdb9323ee2cb5651b28af3b4bf75877"}
Nov 02 09:50:29.401978 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.401189    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25e7beea7effd4df14500c2aeae7aa34cbdb9323ee2cb5651b28af3b4bf75877"
Nov 02 09:50:29.431757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.429414    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:50:29.434761 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.434730    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.580938    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-envoy\") pod \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.581014    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-credential-socket\") pod \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.581066    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-podinfo\") pod \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.581104    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istiod-ca-cert\") pod \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.581156    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-podinfo\") pod \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.581303    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-certs\") pod \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.581360    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-envoy\") pod \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.581429    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-data\") pod \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.581480    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istiod-ca-cert\") pod \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.581515    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-certs\") pod \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.581550    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-kube-api-access-bbrwk\" (UniqueName: \"kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-kube-api-access-bbrwk\") pod \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.581633    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-token\" (UniqueName: \"kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-token\") pod \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.582274    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-data" pod "60302c85-67c9-40da-b62c-1ff3ffdbcf1f" (UID: "60302c85-67c9-40da-b62c-1ff3ffdbcf1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.582580    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-certs" pod "dd0a870c-883a-4314-9e4f-d05fb3241d5a" (UID: "dd0a870c-883a-4314-9e4f-d05fb3241d5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.583078    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-credential-socket" pod "60302c85-67c9-40da-b62c-1ff3ffdbcf1f" (UID: "60302c85-67c9-40da-b62c-1ff3ffdbcf1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.584043    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-certs" pod "60302c85-67c9-40da-b62c-1ff3ffdbcf1f" (UID: "60302c85-67c9-40da-b62c-1ff3ffdbcf1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.584968    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istiod-ca-cert" pod "60302c85-67c9-40da-b62c-1ff3ffdbcf1f" (UID: "60302c85-67c9-40da-b62c-1ff3ffdbcf1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.585277    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-socket\") pod \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.585350    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-socket\") pod \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.585399    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-kube-api-access-gktpd\" (UniqueName: \"kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-kube-api-access-gktpd\") pod \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.585432    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-credential-socket\") pod \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.585467    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-data\") pod \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\" (UID: \"dd0a870c-883a-4314-9e4f-d05fb3241d5a\") "
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.585508    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istiod-ca-cert" pod "dd0a870c-883a-4314-9e4f-d05fb3241d5a" (UID: "dd0a870c-883a-4314-9e4f-d05fb3241d5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.585735    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-socket" pod "60302c85-67c9-40da-b62c-1ff3ffdbcf1f" (UID: "60302c85-67c9-40da-b62c-1ff3ffdbcf1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.590850    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.590876    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.590892    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.592251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.590906    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.636325 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.608309    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-socket" pod "dd0a870c-883a-4314-9e4f-d05fb3241d5a" (UID: "dd0a870c-883a-4314-9e4f-d05fb3241d5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:29.640012 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.639977    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-data" pod "dd0a870c-883a-4314-9e4f-d05fb3241d5a" (UID: "dd0a870c-883a-4314-9e4f-d05fb3241d5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:29.641717 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.641644    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-credential-socket" pod "dd0a870c-883a-4314-9e4f-d05fb3241d5a" (UID: "dd0a870c-883a-4314-9e4f-d05fb3241d5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:29.641903 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.590921    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.641903 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.641755    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.658519 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.654884    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.658519 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.644400    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-kube-api-access-bbrwk" pod "60302c85-67c9-40da-b62c-1ff3ffdbcf1f" (UID: "60302c85-67c9-40da-b62c-1ff3ffdbcf1f"). InnerVolumeSpecName "kube-api-access-bbrwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:29.679871 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.679757    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-podinfo" pod "dd0a870c-883a-4314-9e4f-d05fb3241d5a" (UID: "dd0a870c-883a-4314-9e4f-d05fb3241d5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:50:29.679871 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.679843    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-podinfo" pod "60302c85-67c9-40da-b62c-1ff3ffdbcf1f" (UID: "60302c85-67c9-40da-b62c-1ff3ffdbcf1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:50:29.695667 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.695611    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-token" pod "dd0a870c-883a-4314-9e4f-d05fb3241d5a" (UID: "dd0a870c-883a-4314-9e4f-d05fb3241d5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:29.696025 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.695987    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-envoy" pod "60302c85-67c9-40da-b62c-1ff3ffdbcf1f" (UID: "60302c85-67c9-40da-b62c-1ff3ffdbcf1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:29.696405 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.696369    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-envoy" pod "dd0a870c-883a-4314-9e4f-d05fb3241d5a" (UID: "dd0a870c-883a-4314-9e4f-d05fb3241d5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:29.700647 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.700609    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-kube-api-access-gktpd" pod "dd0a870c-883a-4314-9e4f-d05fb3241d5a" (UID: "dd0a870c-883a-4314-9e4f-d05fb3241d5a"). InnerVolumeSpecName "kube-api-access-gktpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:29.756478 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.756439    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-token\" (UniqueName: \"kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-token\") pod \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\" (UID: \"60302c85-67c9-40da-b62c-1ff3ffdbcf1f\") "
Nov 02 09:50:29.757142 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.757118    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.757273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.757148    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.757273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.757165    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gktpd\" (UniqueName: \"kubernetes.io/projected/dd0a870c-883a-4314-9e4f-d05fb3241d5a-kube-api-access-gktpd\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.757273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.757182    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.757273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.757198    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.757273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.757230    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.757273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.757252    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.757273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.757267    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.757664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.757281    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0a870c-883a-4314-9e4f-d05fb3241d5a-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.757664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.757297    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbrwk\" (UniqueName: \"kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-kube-api-access-bbrwk\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.759812 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.759780    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-token" pod "60302c85-67c9-40da-b62c-1ff3ffdbcf1f" (UID: "60302c85-67c9-40da-b62c-1ff3ffdbcf1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:29.858527 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.858444    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60302c85-67c9-40da-b62c-1ff3ffdbcf1f-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:29.861858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.861813    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d" podUID="4024dd80-c4b0-4a24-a983-ba34eb385c57" containerName="payload-processing" containerID="cri-o://5df023dce23c1f31fac6da011c97f09adcfc20dcb72fa5567a3994efcea4e436" gracePeriod=25
Nov 02 09:50:29.951129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951082    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"]
Nov 02 09:50:29.951812 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951789    4449 cpu_manager.go:414] "containerMap: removing container" podUID="7ef35a63-382b-4978-a07d-197c7775de78" containerName="istio-proxy"
Nov 02 09:50:29.951932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951813    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef35a63-382b-4978-a07d-197c7775de78" containerName="istio-proxy"
Nov 02 09:50:29.951932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951849    4449 cpu_manager.go:414] "containerMap: removing container" podUID="7ef35a63-382b-4978-a07d-197c7775de78" containerName="istio-proxy"
Nov 02 09:50:29.951932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951857    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef35a63-382b-4978-a07d-197c7775de78" containerName="istio-proxy"
Nov 02 09:50:29.951932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951891    4449 cpu_manager.go:414] "containerMap: removing container" podUID="60302c85-67c9-40da-b62c-1ff3ffdbcf1f" containerName="istio-proxy"
Nov 02 09:50:29.951932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951898    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60302c85-67c9-40da-b62c-1ff3ffdbcf1f" containerName="istio-proxy"
Nov 02 09:50:29.951932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951916    4449 cpu_manager.go:414] "containerMap: removing container" podUID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" containerName="istio-proxy"
Nov 02 09:50:29.951932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951923    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" containerName="istio-proxy"
Nov 02 09:50:29.952310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951936    4449 cpu_manager.go:414] "containerMap: removing container" podUID="79021ba5-953f-461e-b75e-957242a29b3a" containerName="revoke-keys"
Nov 02 09:50:29.952310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951943    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79021ba5-953f-461e-b75e-957242a29b3a" containerName="revoke-keys"
Nov 02 09:50:29.952310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951961    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0c609200-d239-4c5a-8743-acc3d200579f" containerName="payload-processing"
Nov 02 09:50:29.952310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951970    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c609200-d239-4c5a-8743-acc3d200579f" containerName="payload-processing"
Nov 02 09:50:29.952310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951987    4449 cpu_manager.go:414] "containerMap: removing container" podUID="6a361364-cc00-418a-9334-47bbce091b43" containerName="istio-proxy"
Nov 02 09:50:29.952310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.951995    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a361364-cc00-418a-9334-47bbce091b43" containerName="istio-proxy"
Nov 02 09:50:29.952310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.952138    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef35a63-382b-4978-a07d-197c7775de78" containerName="istio-proxy"
Nov 02 09:50:29.952310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.952153    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="60302c85-67c9-40da-b62c-1ff3ffdbcf1f" containerName="istio-proxy"
Nov 02 09:50:29.952310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.952165    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" containerName="istio-proxy"
Nov 02 09:50:29.952310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.952174    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="79021ba5-953f-461e-b75e-957242a29b3a" containerName="revoke-keys"
Nov 02 09:50:29.952310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.952183    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c609200-d239-4c5a-8743-acc3d200579f" containerName="payload-processing"
Nov 02 09:50:29.952310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.952193    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a361364-cc00-418a-9334-47bbce091b43" containerName="istio-proxy"
Nov 02 09:50:29.957611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.957588    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"
Nov 02 09:50:29.962131 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.961495    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:50:29.962131 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.961705    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:50:29.962131 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.961917    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:50:29.962131 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.961979    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:50:29.971826 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:29.971798    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"]
Nov 02 09:50:30.061995 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.061957    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67af8aeb-qtq7g\" (UID: \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"
Nov 02 09:50:30.062335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.062309    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxs7h\" (UniqueName: \"kubernetes.io/projected/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-kube-api-access-lxs7h\") pod \"maas-api-revoke-sub-67af8aeb-qtq7g\" (UID: \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"
Nov 02 09:50:30.062482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.062382    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67af8aeb-qtq7g\" (UID: \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"
Nov 02 09:50:30.163836 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.163749    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxs7h\" (UniqueName: \"kubernetes.io/projected/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-kube-api-access-lxs7h\") pod \"maas-api-revoke-sub-67af8aeb-qtq7g\" (UID: \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"
Nov 02 09:50:30.163836 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.163806    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67af8aeb-qtq7g\" (UID: \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"
Nov 02 09:50:30.164045 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.163875    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67af8aeb-qtq7g\" (UID: \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"
Nov 02 09:50:30.164633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.164604    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67af8aeb-qtq7g\" (UID: \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"
Nov 02 09:50:30.166442 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.166416    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67af8aeb-qtq7g\" (UID: \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"
Nov 02 09:50:30.177772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.177745    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxs7h\" (UniqueName: \"kubernetes.io/projected/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-kube-api-access-lxs7h\") pod \"maas-api-revoke-sub-67af8aeb-qtq7g\" (UID: \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"
Nov 02 09:50:30.232647 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.232606    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d" podUID="4024dd80-c4b0-4a24-a983-ba34eb385c57" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:50:30.279842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.279812    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"
Nov 02 09:50:30.411877 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.411844    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"
Nov 02 09:50:30.412066 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.411848    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446" event={"ID":"dd0a870c-883a-4314-9e4f-d05fb3241d5a","Type":"ContainerDied","Data":"d66cc5c5af721fc8b5e094e172ab28e89dd862b974a436b26be61dc278cb7de0"}
Nov 02 09:50:30.412066 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.411993    4449 scope.go:122] "RemoveContainer" containerID="f12027ca32095dbe15199fb3c209cfb642e4e2251ad89e9011172ea8b10ee92d"
Nov 02 09:50:30.412280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.412195    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"
Nov 02 09:50:30.445331 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.445297    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"]
Nov 02 09:50:30.450472 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.450437    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-f8wtz"]
Nov 02 09:50:30.461240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.461188    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"]
Nov 02 09:50:30.466329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.466285    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-h4446"]
Nov 02 09:50:30.569413 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:30.569382    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbc7ca7_5a25_4c72_af8d_a6f9747b2b41.slice/crio-79d7d736626c57db9708b5e2ff4684af53e0db9ec05594d8580483b2311cf8d8 WatchSource:0}: Error finding container 79d7d736626c57db9708b5e2ff4684af53e0db9ec05594d8580483b2311cf8d8: Status 404 returned error can't find the container with id 79d7d736626c57db9708b5e2ff4684af53e0db9ec05594d8580483b2311cf8d8
Nov 02 09:50:30.570057 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.570034    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"]
Nov 02 09:50:30.652164 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.652125    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"]
Nov 02 09:50:30.652921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.652900    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef35a63-382b-4978-a07d-197c7775de78" containerName="istio-proxy"
Nov 02 09:50:30.658256 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.658233    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"
Nov 02 09:50:30.666921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.666887    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"]
Nov 02 09:50:30.771756 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.771722    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj4cf\" (UniqueName: \"kubernetes.io/projected/10a415d7-2ebb-48ed-a643-550892ef3774-kube-api-access-hj4cf\") pod \"maas-api-revoke-sub-da4a2740-d7p2s\" (UID: \"10a415d7-2ebb-48ed-a643-550892ef3774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"
Nov 02 09:50:30.772100 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.771886    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da4a2740-d7p2s\" (UID: \"10a415d7-2ebb-48ed-a643-550892ef3774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"
Nov 02 09:50:30.772100 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.771920    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da4a2740-d7p2s\" (UID: \"10a415d7-2ebb-48ed-a643-550892ef3774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"
Nov 02 09:50:30.873095 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.873066    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj4cf\" (UniqueName: \"kubernetes.io/projected/10a415d7-2ebb-48ed-a643-550892ef3774-kube-api-access-hj4cf\") pod \"maas-api-revoke-sub-da4a2740-d7p2s\" (UID: \"10a415d7-2ebb-48ed-a643-550892ef3774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"
Nov 02 09:50:30.873279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.873153    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da4a2740-d7p2s\" (UID: \"10a415d7-2ebb-48ed-a643-550892ef3774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"
Nov 02 09:50:30.873279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.873173    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da4a2740-d7p2s\" (UID: \"10a415d7-2ebb-48ed-a643-550892ef3774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"
Nov 02 09:50:30.873810 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.873787    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da4a2740-d7p2s\" (UID: \"10a415d7-2ebb-48ed-a643-550892ef3774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"
Nov 02 09:50:30.875724 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.875702    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da4a2740-d7p2s\" (UID: \"10a415d7-2ebb-48ed-a643-550892ef3774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"
Nov 02 09:50:30.885133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.885108    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj4cf\" (UniqueName: \"kubernetes.io/projected/10a415d7-2ebb-48ed-a643-550892ef3774-kube-api-access-hj4cf\") pod \"maas-api-revoke-sub-da4a2740-d7p2s\" (UID: \"10a415d7-2ebb-48ed-a643-550892ef3774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"
Nov 02 09:50:30.977571 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:30.977541    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"
Nov 02 09:50:31.210783 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:31.210750    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60302c85-67c9-40da-b62c-1ff3ffdbcf1f" path="/var/lib/kubelet/pods/60302c85-67c9-40da-b62c-1ff3ffdbcf1f/volumes"
Nov 02 09:50:31.211368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:31.211351    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0a870c-883a-4314-9e4f-d05fb3241d5a" path="/var/lib/kubelet/pods/dd0a870c-883a-4314-9e4f-d05fb3241d5a/volumes"
Nov 02 09:50:31.269344 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:31.269299    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a415d7_2ebb_48ed_a643_550892ef3774.slice/crio-b2a38e482b15228684d4d7be2c9749b029faa394c81a91366a983b2ab4c56cff WatchSource:0}: Error finding container b2a38e482b15228684d4d7be2c9749b029faa394c81a91366a983b2ab4c56cff: Status 404 returned error can't find the container with id b2a38e482b15228684d4d7be2c9749b029faa394c81a91366a983b2ab4c56cff
Nov 02 09:50:31.269621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:31.269601    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"]
Nov 02 09:50:31.421567 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:31.421532    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g" event={"ID":"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41","Type":"ContainerStarted","Data":"e6fa159e9ac7d42a71fb507394a652c2d82c0b006c021f6651768682d477eb59"}
Nov 02 09:50:31.421754 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:31.421586    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g" event={"ID":"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41","Type":"ContainerStarted","Data":"79d7d736626c57db9708b5e2ff4684af53e0db9ec05594d8580483b2311cf8d8"}
Nov 02 09:50:31.422862 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:31.422833    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s" event={"ID":"10a415d7-2ebb-48ed-a643-550892ef3774","Type":"ContainerStarted","Data":"b2a38e482b15228684d4d7be2c9749b029faa394c81a91366a983b2ab4c56cff"}
Nov 02 09:50:31.442794 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:31.442726    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g" podStartSLOduration=2.442712035 podStartE2EDuration="2.442712035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:31.438985883 +0000 UTC m=+908.377272660" watchObservedRunningTime="2026-11-02 09:50:31.442712035 +0000 UTC m=+908.380998843"
Nov 02 09:50:32.050270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.050237    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"
Nov 02 09:50:32.098659 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.098593    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"
Nov 02 09:50:32.193664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.193631    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094a6ff2-c320-45e3-839a-9ba70ec6da87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094a6ff2-c320-45e3-839a-9ba70ec6da87-plugins-config-volume\") pod \"094a6ff2-c320-45e3-839a-9ba70ec6da87\" (UID: \"094a6ff2-c320-45e3-839a-9ba70ec6da87\") "
Nov 02 09:50:32.193831 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.193684    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094a6ff2-c320-45e3-839a-9ba70ec6da87-kube-api-access-rqsdn\" (UniqueName: \"kubernetes.io/projected/094a6ff2-c320-45e3-839a-9ba70ec6da87-kube-api-access-rqsdn\") pod \"094a6ff2-c320-45e3-839a-9ba70ec6da87\" (UID: \"094a6ff2-c320-45e3-839a-9ba70ec6da87\") "
Nov 02 09:50:32.193831 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.193730    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-kube-api-access-vvcfh\" (UniqueName: \"kubernetes.io/projected/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-kube-api-access-vvcfh\") pod \"f1a365ff-43d5-400d-80d6-0cd798f2f7bf\" (UID: \"f1a365ff-43d5-400d-80d6-0cd798f2f7bf\") "
Nov 02 09:50:32.193923 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.193855    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-plugins-config-volume\") pod \"f1a365ff-43d5-400d-80d6-0cd798f2f7bf\" (UID: \"f1a365ff-43d5-400d-80d6-0cd798f2f7bf\") "
Nov 02 09:50:32.194210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.194179    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094a6ff2-c320-45e3-839a-9ba70ec6da87-plugins-config-volume" pod "094a6ff2-c320-45e3-839a-9ba70ec6da87" (UID: "094a6ff2-c320-45e3-839a-9ba70ec6da87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:32.194318 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.194207    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-plugins-config-volume" pod "f1a365ff-43d5-400d-80d6-0cd798f2f7bf" (UID: "f1a365ff-43d5-400d-80d6-0cd798f2f7bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:32.194494 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.194477    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:32.194541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.194496    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094a6ff2-c320-45e3-839a-9ba70ec6da87-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:32.195984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.195964    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094a6ff2-c320-45e3-839a-9ba70ec6da87-kube-api-access-rqsdn" pod "094a6ff2-c320-45e3-839a-9ba70ec6da87" (UID: "094a6ff2-c320-45e3-839a-9ba70ec6da87"). InnerVolumeSpecName "kube-api-access-rqsdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:32.196043 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.195992    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-kube-api-access-vvcfh" pod "f1a365ff-43d5-400d-80d6-0cd798f2f7bf" (UID: "f1a365ff-43d5-400d-80d6-0cd798f2f7bf"). InnerVolumeSpecName "kube-api-access-vvcfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:32.296489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.296446    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqsdn\" (UniqueName: \"kubernetes.io/projected/094a6ff2-c320-45e3-839a-9ba70ec6da87-kube-api-access-rqsdn\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:32.296489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.296485    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvcfh\" (UniqueName: \"kubernetes.io/projected/f1a365ff-43d5-400d-80d6-0cd798f2f7bf-kube-api-access-vvcfh\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:32.434020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.433933    4449 generic.go:356] "Generic (PLEG): container finished" podID="0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41" containerID="e6fa159e9ac7d42a71fb507394a652c2d82c0b006c021f6651768682d477eb59" exitCode=0
Nov 02 09:50:32.434188 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.434016    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g" event={"ID":"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41","Type":"ContainerDied","Data":"e6fa159e9ac7d42a71fb507394a652c2d82c0b006c021f6651768682d477eb59"}
Nov 02 09:50:32.435934 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.435910    4449 generic.go:356] "Generic (PLEG): container finished" podID="10a415d7-2ebb-48ed-a643-550892ef3774" containerID="0c1d2e8b3829b57a120a6709ec71ed4e9232c61a5f8fd84f8ad47b0998e54956" exitCode=0
Nov 02 09:50:32.436057 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.436033    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s" event={"ID":"10a415d7-2ebb-48ed-a643-550892ef3774","Type":"ContainerDied","Data":"0c1d2e8b3829b57a120a6709ec71ed4e9232c61a5f8fd84f8ad47b0998e54956"}
Nov 02 09:50:32.437579 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.437558    4449 generic.go:356] "Generic (PLEG): container finished" podID="f1a365ff-43d5-400d-80d6-0cd798f2f7bf" containerID="62f07ce53eadb44da1b295ba773b1cc43769adde6b287a12278dee3f7030a3b4" exitCode=0
Nov 02 09:50:32.437689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.437602    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"
Nov 02 09:50:32.437689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.437620    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4" event={"ID":"f1a365ff-43d5-400d-80d6-0cd798f2f7bf","Type":"ContainerDied","Data":"62f07ce53eadb44da1b295ba773b1cc43769adde6b287a12278dee3f7030a3b4"}
Nov 02 09:50:32.437689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.437663    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4" event={"ID":"f1a365ff-43d5-400d-80d6-0cd798f2f7bf","Type":"ContainerDied","Data":"20a596f02629fc1a3e29f169de1fa7c88230088e45b2a16a21f4bf933c3b816c"}
Nov 02 09:50:32.437863 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.437688    4449 scope.go:122] "RemoveContainer" containerID="62f07ce53eadb44da1b295ba773b1cc43769adde6b287a12278dee3f7030a3b4"
Nov 02 09:50:32.439408 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.439385    4449 generic.go:356] "Generic (PLEG): container finished" podID="094a6ff2-c320-45e3-839a-9ba70ec6da87" containerID="9064bddc66b23dfde817a3367c46c126568525071e56f8f23799b1632f6149d2" exitCode=0
Nov 02 09:50:32.439517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.439426    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"
Nov 02 09:50:32.439517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.439430    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll" event={"ID":"094a6ff2-c320-45e3-839a-9ba70ec6da87","Type":"ContainerDied","Data":"9064bddc66b23dfde817a3367c46c126568525071e56f8f23799b1632f6149d2"}
Nov 02 09:50:32.439517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.439481    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll" event={"ID":"094a6ff2-c320-45e3-839a-9ba70ec6da87","Type":"ContainerDied","Data":"16ee763ca41fb4b6a7ff235fd9714d6ab6d545795b19b4de917bbbeca8085ca7"}
Nov 02 09:50:32.449281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.449159    4449 scope.go:122] "RemoveContainer" containerID="62f07ce53eadb44da1b295ba773b1cc43769adde6b287a12278dee3f7030a3b4"
Nov 02 09:50:32.449661 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:50:32.449641    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f07ce53eadb44da1b295ba773b1cc43769adde6b287a12278dee3f7030a3b4\": container with ID starting with 62f07ce53eadb44da1b295ba773b1cc43769adde6b287a12278dee3f7030a3b4 not found: ID does not exist" containerID="62f07ce53eadb44da1b295ba773b1cc43769adde6b287a12278dee3f7030a3b4"
Nov 02 09:50:32.449735 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.449672    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f07ce53eadb44da1b295ba773b1cc43769adde6b287a12278dee3f7030a3b4"} err="failed to get container status \"62f07ce53eadb44da1b295ba773b1cc43769adde6b287a12278dee3f7030a3b4\": rpc error: code = NotFound desc = could not find container \"62f07ce53eadb44da1b295ba773b1cc43769adde6b287a12278dee3f7030a3b4\": container with ID starting with 62f07ce53eadb44da1b295ba773b1cc43769adde6b287a12278dee3f7030a3b4 not found: ID does not exist"
Nov 02 09:50:32.449735 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.449692    4449 scope.go:122] "RemoveContainer" containerID="9064bddc66b23dfde817a3367c46c126568525071e56f8f23799b1632f6149d2"
Nov 02 09:50:32.463675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.463647    4449 scope.go:122] "RemoveContainer" containerID="9064bddc66b23dfde817a3367c46c126568525071e56f8f23799b1632f6149d2"
Nov 02 09:50:32.464148 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:50:32.464114    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9064bddc66b23dfde817a3367c46c126568525071e56f8f23799b1632f6149d2\": container with ID starting with 9064bddc66b23dfde817a3367c46c126568525071e56f8f23799b1632f6149d2 not found: ID does not exist" containerID="9064bddc66b23dfde817a3367c46c126568525071e56f8f23799b1632f6149d2"
Nov 02 09:50:32.464278 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.464159    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9064bddc66b23dfde817a3367c46c126568525071e56f8f23799b1632f6149d2"} err="failed to get container status \"9064bddc66b23dfde817a3367c46c126568525071e56f8f23799b1632f6149d2\": rpc error: code = NotFound desc = could not find container \"9064bddc66b23dfde817a3367c46c126568525071e56f8f23799b1632f6149d2\": container with ID starting with 9064bddc66b23dfde817a3367c46c126568525071e56f8f23799b1632f6149d2 not found: ID does not exist"
Nov 02 09:50:32.473955 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.473924    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"]
Nov 02 09:50:32.478756 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.478728    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-301be8-74b7d9d5ff-5xjll"]
Nov 02 09:50:32.490452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.490425    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"]
Nov 02 09:50:32.497133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:32.497109    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-301be8-9687dd9ff-n9tq4"]
Nov 02 09:50:33.208052 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.208013    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094a6ff2-c320-45e3-839a-9ba70ec6da87" path="/var/lib/kubelet/pods/094a6ff2-c320-45e3-839a-9ba70ec6da87/volumes"
Nov 02 09:50:33.208707 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.208684    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a365ff-43d5-400d-80d6-0cd798f2f7bf" path="/var/lib/kubelet/pods/f1a365ff-43d5-400d-80d6-0cd798f2f7bf/volumes"
Nov 02 09:50:33.243828 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.243792    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"]
Nov 02 09:50:33.244930 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.244850    4449 cpu_manager.go:414] "containerMap: removing container" podUID="094a6ff2-c320-45e3-839a-9ba70ec6da87" containerName="payload-processing"
Nov 02 09:50:33.244930 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.244875    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094a6ff2-c320-45e3-839a-9ba70ec6da87" containerName="payload-processing"
Nov 02 09:50:33.244930 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.244900    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f1a365ff-43d5-400d-80d6-0cd798f2f7bf" containerName="payload-pre-processing"
Nov 02 09:50:33.244930 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.244908    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a365ff-43d5-400d-80d6-0cd798f2f7bf" containerName="payload-pre-processing"
Nov 02 09:50:33.245243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.245090    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="094a6ff2-c320-45e3-839a-9ba70ec6da87" containerName="payload-processing"
Nov 02 09:50:33.245243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.245108    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a365ff-43d5-400d-80d6-0cd798f2f7bf" containerName="payload-pre-processing"
Nov 02 09:50:33.251054 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.251015    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.258626 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.258594    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"]
Nov 02 09:50:33.411839 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.411807    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxf9r\" (UniqueName: \"kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-kube-api-access-kxf9r\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.411994 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.411902    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.411994 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.411937    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.411994 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.411972    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.412123 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.412057    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.412123 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.412113    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a51c35-655c-4e06-8524-e19e99eb48f1-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.412201 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.412190    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.412290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.412273    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.412343 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.412327    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.516291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.516179    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.516457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.516300    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.516457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.516374    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a51c35-655c-4e06-8524-e19e99eb48f1-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.516529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.516494    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.516581 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.516561    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.516617 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.516599    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.516709 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.516695    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxf9r\" (UniqueName: \"kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-kube-api-access-kxf9r\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.516752 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.516734    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.516797 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.516763    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.517638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.517610    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.518436 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.518387    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.518436 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.518408    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.518632 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.518606    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a51c35-655c-4e06-8524-e19e99eb48f1-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.518824 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.518789    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.522627 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.522597    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.527156 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.527086    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.540858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.540146    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.557463 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.557362    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxf9r\" (UniqueName: \"kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-kube-api-access-kxf9r\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.568441 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.568405    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:33.641843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.641811    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"
Nov 02 09:50:33.674540 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.674515    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"
Nov 02 09:50:33.728253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.728197    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-openshift-ca\") pod \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\" (UID: \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\") "
Nov 02 09:50:33.728407 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.728386    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-xks-ca\") pod \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\" (UID: \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\") "
Nov 02 09:50:33.728540 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.728524    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-kube-api-access-lxs7h\" (UniqueName: \"kubernetes.io/projected/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-kube-api-access-lxs7h\") pod \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\" (UID: \"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41\") "
Nov 02 09:50:33.728662 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.728622    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-openshift-ca" pod "0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41" (UID: "0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:33.729027 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.729002    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:33.730777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.730753    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-xks-ca" pod "0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41" (UID: "0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:50:33.730858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.730791    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-kube-api-access-lxs7h" pod "0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41" (UID: "0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41"). InnerVolumeSpecName "kube-api-access-lxs7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:33.829571 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.829546    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-xks-ca\") pod \"10a415d7-2ebb-48ed-a643-550892ef3774\" (UID: \"10a415d7-2ebb-48ed-a643-550892ef3774\") "
Nov 02 09:50:33.829746 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.829595    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-openshift-ca\") pod \"10a415d7-2ebb-48ed-a643-550892ef3774\" (UID: \"10a415d7-2ebb-48ed-a643-550892ef3774\") "
Nov 02 09:50:33.829746 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.829638    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a415d7-2ebb-48ed-a643-550892ef3774-kube-api-access-hj4cf\" (UniqueName: \"kubernetes.io/projected/10a415d7-2ebb-48ed-a643-550892ef3774-kube-api-access-hj4cf\") pod \"10a415d7-2ebb-48ed-a643-550892ef3774\" (UID: \"10a415d7-2ebb-48ed-a643-550892ef3774\") "
Nov 02 09:50:33.830024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.829997    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-openshift-ca" pod "10a415d7-2ebb-48ed-a643-550892ef3774" (UID: "10a415d7-2ebb-48ed-a643-550892ef3774"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:33.830102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.830046    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:33.830102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.830062    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxs7h\" (UniqueName: \"kubernetes.io/projected/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41-kube-api-access-lxs7h\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:33.832170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.832143    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-xks-ca" pod "10a415d7-2ebb-48ed-a643-550892ef3774" (UID: "10a415d7-2ebb-48ed-a643-550892ef3774"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:50:33.832341 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.832318    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a415d7-2ebb-48ed-a643-550892ef3774-kube-api-access-hj4cf" pod "10a415d7-2ebb-48ed-a643-550892ef3774" (UID: "10a415d7-2ebb-48ed-a643-550892ef3774"). InnerVolumeSpecName "kube-api-access-hj4cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:33.869242 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:33.869201    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a51c35_655c_4e06_8524_e19e99eb48f1.slice/crio-e418bcb4bb90fe10fd957308f20a97aafcc860103525dd2af573523116862910 WatchSource:0}: Error finding container e418bcb4bb90fe10fd957308f20a97aafcc860103525dd2af573523116862910: Status 404 returned error can't find the container with id e418bcb4bb90fe10fd957308f20a97aafcc860103525dd2af573523116862910
Nov 02 09:50:33.869466 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.869445    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"]
Nov 02 09:50:33.871620 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.871589    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:50:33.871687 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.871668    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:50:33.871723 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.871696    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:50:33.931987 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.931964    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:33.932102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.931993    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a415d7-2ebb-48ed-a643-550892ef3774-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:33.932102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:33.932011    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj4cf\" (UniqueName: \"kubernetes.io/projected/10a415d7-2ebb-48ed-a643-550892ef3774-kube-api-access-hj4cf\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:34.482761 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:34.482731    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"
Nov 02 09:50:34.483255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:34.482764    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s" event={"ID":"10a415d7-2ebb-48ed-a643-550892ef3774","Type":"ContainerDied","Data":"b2a38e482b15228684d4d7be2c9749b029faa394c81a91366a983b2ab4c56cff"}
Nov 02 09:50:34.483255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:34.482816    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2a38e482b15228684d4d7be2c9749b029faa394c81a91366a983b2ab4c56cff"
Nov 02 09:50:34.485053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:34.484964    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb" event={"ID":"a0a51c35-655c-4e06-8524-e19e99eb48f1","Type":"ContainerStarted","Data":"0dbb3ab956287fc15110c0cac978afe02e7804bcf581ed6ca653a43d5ba43905"}
Nov 02 09:50:34.485053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:34.485020    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb" event={"ID":"a0a51c35-655c-4e06-8524-e19e99eb48f1","Type":"ContainerStarted","Data":"e418bcb4bb90fe10fd957308f20a97aafcc860103525dd2af573523116862910"}
Nov 02 09:50:34.486783 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:34.486759    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g" event={"ID":"0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41","Type":"ContainerDied","Data":"79d7d736626c57db9708b5e2ff4684af53e0db9ec05594d8580483b2311cf8d8"}
Nov 02 09:50:34.486881 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:34.486795    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79d7d736626c57db9708b5e2ff4684af53e0db9ec05594d8580483b2311cf8d8"
Nov 02 09:50:34.486881 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:34.486794    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"
Nov 02 09:50:34.510375 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:34.510330    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb" podStartSLOduration=1.510315007 podStartE2EDuration="1.510315007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:34.506414039 +0000 UTC m=+911.444700834" watchObservedRunningTime="2026-11-02 09:50:34.510315007 +0000 UTC m=+911.448601882"
Nov 02 09:50:34.574530 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:34.574486    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:34.576298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:34.576268    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 09:50:34.576457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:34.576319    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb" podUID="a0a51c35-655c-4e06-8524-e19e99eb48f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 09:50:35.572665 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:35.572623    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 09:50:35.573269 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:35.573204    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb" podUID="a0a51c35-655c-4e06-8524-e19e99eb48f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 09:50:36.333170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.333125    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"]
Nov 02 09:50:36.333795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.333782    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41" containerName="revoke-keys"
Nov 02 09:50:36.333835 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.333797    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41" containerName="revoke-keys"
Nov 02 09:50:36.333835 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.333808    4449 cpu_manager.go:414] "containerMap: removing container" podUID="10a415d7-2ebb-48ed-a643-550892ef3774" containerName="revoke-keys"
Nov 02 09:50:36.333835 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.333814    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a415d7-2ebb-48ed-a643-550892ef3774" containerName="revoke-keys"
Nov 02 09:50:36.333981 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.333896    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41" containerName="revoke-keys"
Nov 02 09:50:36.333981 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.333904    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a415d7-2ebb-48ed-a643-550892ef3774" containerName="revoke-keys"
Nov 02 09:50:36.337800 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.337784    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.344734 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.344699    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"openshift-service-ca.crt\""
Nov 02 09:50:36.347476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.346174    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-32e810beae6e\""
Nov 02 09:50:36.347476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.346287    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"routing-overlay\""
Nov 02 09:50:36.347476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.346393    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-e2e--e7669bbb-dockercfg-zngpv\""
Nov 02 09:50:36.347476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.346437    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"kube-root-ca.crt\""
Nov 02 09:50:36.347476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.346510    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-plugins-e2e-extmod-b-204fba\""
Nov 02 09:50:36.347476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.346559    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-4a99520c97db\""
Nov 02 09:50:36.347476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.346593    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-11e3267f4507\""
Nov 02 09:50:36.347476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.346640    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-1521404c1537\""
Nov 02 09:50:36.347476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.346709    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-8960fbffc2fc\""
Nov 02 09:50:36.347476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.347340    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"]
Nov 02 09:50:36.485663 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.485617    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.485857 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.485782    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gss79\" (UniqueName: \"kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-kube-api-access-gss79\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.485943 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.485928    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.486037 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.486023    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.571250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.570590    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 09:50:36.571250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.570651    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb" podUID="a0a51c35-655c-4e06-8524-e19e99eb48f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 09:50:36.584518 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:50:36.584411    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:50:36.587320 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.587295    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.587437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.587374    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gss79\" (UniqueName: \"kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-kube-api-access-gss79\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.587501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.587483    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.587576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.587560    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.588368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.588343    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.588698 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.588675    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.590745 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.590725    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.600206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.600186    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gss79\" (UniqueName: \"kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-kube-api-access-gss79\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.684442 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.684405    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:36.979664 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:36.979632    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb1e8a4_75d1_4b42_995c_089ab025f1c1.slice/crio-5a0503958e0ae1321eefad63ff10dfc007c041d76f33da23fc6c4afa8243898f WatchSource:0}: Error finding container 5a0503958e0ae1321eefad63ff10dfc007c041d76f33da23fc6c4afa8243898f: Status 404 returned error can't find the container with id 5a0503958e0ae1321eefad63ff10dfc007c041d76f33da23fc6c4afa8243898f
Nov 02 09:50:36.981965 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:36.981939    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"]
Nov 02 09:50:37.519033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:37.518995    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr" event={"ID":"1bb1e8a4-75d1-4b42-995c-089ab025f1c1","Type":"ContainerStarted","Data":"399a06c4f0bca63a64307bad4ed82c869c2e995fb773040e29fc7ad75f11c761"}
Nov 02 09:50:37.519289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:37.519057    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr" event={"ID":"1bb1e8a4-75d1-4b42-995c-089ab025f1c1","Type":"ContainerStarted","Data":"5a0503958e0ae1321eefad63ff10dfc007c041d76f33da23fc6c4afa8243898f"}
Nov 02 09:50:37.519289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:37.519087    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:37.576681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:37.576641    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 09:50:37.576866 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:37.576702    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb" podUID="a0a51c35-655c-4e06-8524-e19e99eb48f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 09:50:37.581954 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:37.581890    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr" podStartSLOduration=1.5818761989999999 podStartE2EDuration="1.581876199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:37.548584097 +0000 UTC m=+914.486870893" watchObservedRunningTime="2026-11-02 09:50:37.581876199 +0000 UTC m=+914.520162996"
Nov 02 09:50:38.573749 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:38.573705    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 09:50:38.574390 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:38.574341    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb" podUID="a0a51c35-655c-4e06-8524-e19e99eb48f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 09:50:39.193731 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:39.193655    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"]
Nov 02 09:50:39.194251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:39.194192    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" podUID="5928113f-5ce3-4403-b390-d52a356939af" containerName="istio-proxy" containerID="cri-o://67d2ecf39dfa86c0ad1e4a3f5beff9205d0ffe4d3685994c76dd4c045b0f88d0" gracePeriod=30
Nov 02 09:50:39.574249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:39.571693    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 09:50:39.574249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:39.571771    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb" podUID="a0a51c35-655c-4e06-8524-e19e99eb48f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 09:50:40.232636 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:40.232174    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d" podUID="4024dd80-c4b0-4a24-a983-ba34eb385c57" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:50:40.992245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:40.985586    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:40.992245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:40.987168    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:41.006721 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:41.006640    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 09:50:41.748443 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:41.748395    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"]
Nov 02 09:50:41.834245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:41.808544    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"
Nov 02 09:50:41.834245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:41.829585    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"]
Nov 02 09:50:41.834245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:41.832769    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:50:41.834245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:41.832835    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:50:41.834245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:41.832962    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:50:41.834245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:41.833024    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:50:41.933712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:41.933663    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0de25fa-jbpcm\" (UID: \"25edb50b-6bac-4c2d-9045-f1be89706a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"
Nov 02 09:50:41.933915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:41.933718    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cwmj\" (UniqueName: \"kubernetes.io/projected/25edb50b-6bac-4c2d-9045-f1be89706a0b-kube-api-access-5cwmj\") pod \"maas-api-revoke-sub-d0de25fa-jbpcm\" (UID: \"25edb50b-6bac-4c2d-9045-f1be89706a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"
Nov 02 09:50:41.934290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:41.934196    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0de25fa-jbpcm\" (UID: \"25edb50b-6bac-4c2d-9045-f1be89706a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"
Nov 02 09:50:42.035073 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:42.035037    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0de25fa-jbpcm\" (UID: \"25edb50b-6bac-4c2d-9045-f1be89706a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"
Nov 02 09:50:42.035472 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:42.035156    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0de25fa-jbpcm\" (UID: \"25edb50b-6bac-4c2d-9045-f1be89706a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"
Nov 02 09:50:42.035472 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:42.035176    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cwmj\" (UniqueName: \"kubernetes.io/projected/25edb50b-6bac-4c2d-9045-f1be89706a0b-kube-api-access-5cwmj\") pod \"maas-api-revoke-sub-d0de25fa-jbpcm\" (UID: \"25edb50b-6bac-4c2d-9045-f1be89706a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"
Nov 02 09:50:42.035913 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:42.035893    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0de25fa-jbpcm\" (UID: \"25edb50b-6bac-4c2d-9045-f1be89706a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"
Nov 02 09:50:42.037682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:42.037648    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0de25fa-jbpcm\" (UID: \"25edb50b-6bac-4c2d-9045-f1be89706a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"
Nov 02 09:50:42.051295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:42.051265    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cwmj\" (UniqueName: \"kubernetes.io/projected/25edb50b-6bac-4c2d-9045-f1be89706a0b-kube-api-access-5cwmj\") pod \"maas-api-revoke-sub-d0de25fa-jbpcm\" (UID: \"25edb50b-6bac-4c2d-9045-f1be89706a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"
Nov 02 09:50:42.196792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:42.196755    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"
Nov 02 09:50:42.485688 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:42.485640    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25edb50b_6bac_4c2d_9045_f1be89706a0b.slice/crio-c45f6562496e9c5616c0ff16f9c484c051383639c2c2eb3ac1bbb9c263b661f6 WatchSource:0}: Error finding container c45f6562496e9c5616c0ff16f9c484c051383639c2c2eb3ac1bbb9c263b661f6: Status 404 returned error can't find the container with id c45f6562496e9c5616c0ff16f9c484c051383639c2c2eb3ac1bbb9c263b661f6
Nov 02 09:50:42.487701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:42.487672    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"]
Nov 02 09:50:42.655984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:42.655909    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm" event={"ID":"25edb50b-6bac-4c2d-9045-f1be89706a0b","Type":"ContainerStarted","Data":"c45f6562496e9c5616c0ff16f9c484c051383639c2c2eb3ac1bbb9c263b661f6"}
Nov 02 09:50:43.666171 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:43.666131    4449 generic.go:356] "Generic (PLEG): container finished" podID="25edb50b-6bac-4c2d-9045-f1be89706a0b" containerID="e2928c1c0db4e46d93446c4b0c455b61df2fd356182ebb9c194a07036663d522" exitCode=0
Nov 02 09:50:43.666577 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:43.666253    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm" event={"ID":"25edb50b-6bac-4c2d-9045-f1be89706a0b","Type":"ContainerDied","Data":"e2928c1c0db4e46d93446c4b0c455b61df2fd356182ebb9c194a07036663d522"}
Nov 02 09:50:44.499914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.499882    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:50:44.571420 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.571384    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-credential-socket\") pod \"5928113f-5ce3-4403-b390-d52a356939af\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") "
Nov 02 09:50:44.571588 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.571501    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5928113f-5ce3-4403-b390-d52a356939af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5928113f-5ce3-4403-b390-d52a356939af-istio-podinfo\") pod \"5928113f-5ce3-4403-b390-d52a356939af\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") "
Nov 02 09:50:44.571588 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.571558    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-data\") pod \"5928113f-5ce3-4403-b390-d52a356939af\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") "
Nov 02 09:50:44.571588 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.571584    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-envoy\") pod \"5928113f-5ce3-4403-b390-d52a356939af\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") "
Nov 02 09:50:44.571761 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.571727    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-credential-socket" pod "5928113f-5ce3-4403-b390-d52a356939af" (UID: "5928113f-5ce3-4403-b390-d52a356939af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:44.571820 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.571786    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-istio-token\" (UniqueName: \"kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-istio-token\") pod \"5928113f-5ce3-4403-b390-d52a356939af\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") "
Nov 02 09:50:44.571904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.571885    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5928113f-5ce3-4403-b390-d52a356939af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5928113f-5ce3-4403-b390-d52a356939af-istiod-ca-cert\") pod \"5928113f-5ce3-4403-b390-d52a356939af\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") "
Nov 02 09:50:44.571965 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.571908    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-data" pod "5928113f-5ce3-4403-b390-d52a356939af" (UID: "5928113f-5ce3-4403-b390-d52a356939af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:44.571965 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.571958    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-socket\") pod \"5928113f-5ce3-4403-b390-d52a356939af\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") "
Nov 02 09:50:44.572075 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.571995    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-kube-api-access-d4nqj\" (UniqueName: \"kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-kube-api-access-d4nqj\") pod \"5928113f-5ce3-4403-b390-d52a356939af\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") "
Nov 02 09:50:44.572122 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.572086    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-certs\") pod \"5928113f-5ce3-4403-b390-d52a356939af\" (UID: \"5928113f-5ce3-4403-b390-d52a356939af\") "
Nov 02 09:50:44.572799 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.572774    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.572799 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.572800    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.572998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.572979    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-certs" pod "5928113f-5ce3-4403-b390-d52a356939af" (UID: "5928113f-5ce3-4403-b390-d52a356939af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:44.573124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.573027    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5928113f-5ce3-4403-b390-d52a356939af-istiod-ca-cert" pod "5928113f-5ce3-4403-b390-d52a356939af" (UID: "5928113f-5ce3-4403-b390-d52a356939af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:44.573124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.573092    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-socket" pod "5928113f-5ce3-4403-b390-d52a356939af" (UID: "5928113f-5ce3-4403-b390-d52a356939af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:44.578586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.578532    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-istio-token" pod "5928113f-5ce3-4403-b390-d52a356939af" (UID: "5928113f-5ce3-4403-b390-d52a356939af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:44.578586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.578538    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5928113f-5ce3-4403-b390-d52a356939af-istio-podinfo" pod "5928113f-5ce3-4403-b390-d52a356939af" (UID: "5928113f-5ce3-4403-b390-d52a356939af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:50:44.578586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.578527    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-envoy" pod "5928113f-5ce3-4403-b390-d52a356939af" (UID: "5928113f-5ce3-4403-b390-d52a356939af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:44.580664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.580392    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-kube-api-access-d4nqj" pod "5928113f-5ce3-4403-b390-d52a356939af" (UID: "5928113f-5ce3-4403-b390-d52a356939af"). InnerVolumeSpecName "kube-api-access-d4nqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:44.662713 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.662674    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"]
Nov 02 09:50:44.663363 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.663349    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5928113f-5ce3-4403-b390-d52a356939af" containerName="istio-proxy"
Nov 02 09:50:44.663403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.663365    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5928113f-5ce3-4403-b390-d52a356939af" containerName="istio-proxy"
Nov 02 09:50:44.663403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.663377    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5928113f-5ce3-4403-b390-d52a356939af" containerName="istio-proxy"
Nov 02 09:50:44.663403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.663382    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5928113f-5ce3-4403-b390-d52a356939af" containerName="istio-proxy"
Nov 02 09:50:44.663509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.663468    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5928113f-5ce3-4403-b390-d52a356939af" containerName="istio-proxy"
Nov 02 09:50:44.663509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.663480    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5928113f-5ce3-4403-b390-d52a356939af" containerName="istio-proxy"
Nov 02 09:50:44.667410 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.667387    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"
Nov 02 09:50:44.676029 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.675992    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb8wd\" (UniqueName: \"kubernetes.io/projected/981a9928-8e77-4f76-96a1-59f1e541c65e-kube-api-access-nb8wd\") pod \"maas-api-revoke-sub-bf0004c2-kn2lx\" (UID: \"981a9928-8e77-4f76-96a1-59f1e541c65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"
Nov 02 09:50:44.676190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.676172    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0004c2-kn2lx\" (UID: \"981a9928-8e77-4f76-96a1-59f1e541c65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"
Nov 02 09:50:44.676308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.676259    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0004c2-kn2lx\" (UID: \"981a9928-8e77-4f76-96a1-59f1e541c65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"
Nov 02 09:50:44.676428 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.676402    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5928113f-5ce3-4403-b390-d52a356939af-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.676482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.676433    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.676482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.676449    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.676482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.676463    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5928113f-5ce3-4403-b390-d52a356939af-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.676482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.676479    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.676604 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.676494    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4nqj\" (UniqueName: \"kubernetes.io/projected/5928113f-5ce3-4403-b390-d52a356939af-kube-api-access-d4nqj\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.676604 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.676509    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5928113f-5ce3-4403-b390-d52a356939af-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.677693 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.677663    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"]
Nov 02 09:50:44.682265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.682239    4449 generic.go:356] "Generic (PLEG): container finished" podID="5928113f-5ce3-4403-b390-d52a356939af" containerID="67d2ecf39dfa86c0ad1e4a3f5beff9205d0ffe4d3685994c76dd4c045b0f88d0" exitCode=0
Nov 02 09:50:44.682559 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.682545    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"
Nov 02 09:50:44.685504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.684672    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" event={"ID":"5928113f-5ce3-4403-b390-d52a356939af","Type":"ContainerDied","Data":"67d2ecf39dfa86c0ad1e4a3f5beff9205d0ffe4d3685994c76dd4c045b0f88d0"}
Nov 02 09:50:44.685504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.684736    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6" event={"ID":"5928113f-5ce3-4403-b390-d52a356939af","Type":"ContainerDied","Data":"464837686f2ed64a6ac24f12ea5ca26dc1819159c8c8e5aed284c109a033e895"}
Nov 02 09:50:44.685504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.684783    4449 scope.go:122] "RemoveContainer" containerID="67d2ecf39dfa86c0ad1e4a3f5beff9205d0ffe4d3685994c76dd4c045b0f88d0"
Nov 02 09:50:44.699138 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.699110    4449 scope.go:122] "RemoveContainer" containerID="d962a00c6aa93312933d4291fdbf5c290952111d56e3b4657c157b35e7af97c7"
Nov 02 09:50:44.715427 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.715367    4449 scope.go:122] "RemoveContainer" containerID="67d2ecf39dfa86c0ad1e4a3f5beff9205d0ffe4d3685994c76dd4c045b0f88d0"
Nov 02 09:50:44.715817 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:50:44.715792    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d2ecf39dfa86c0ad1e4a3f5beff9205d0ffe4d3685994c76dd4c045b0f88d0\": container with ID starting with 67d2ecf39dfa86c0ad1e4a3f5beff9205d0ffe4d3685994c76dd4c045b0f88d0 not found: ID does not exist" containerID="67d2ecf39dfa86c0ad1e4a3f5beff9205d0ffe4d3685994c76dd4c045b0f88d0"
Nov 02 09:50:44.715935 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.715831    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d2ecf39dfa86c0ad1e4a3f5beff9205d0ffe4d3685994c76dd4c045b0f88d0"} err="failed to get container status \"67d2ecf39dfa86c0ad1e4a3f5beff9205d0ffe4d3685994c76dd4c045b0f88d0\": rpc error: code = NotFound desc = could not find container \"67d2ecf39dfa86c0ad1e4a3f5beff9205d0ffe4d3685994c76dd4c045b0f88d0\": container with ID starting with 67d2ecf39dfa86c0ad1e4a3f5beff9205d0ffe4d3685994c76dd4c045b0f88d0 not found: ID does not exist"
Nov 02 09:50:44.715935 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.715856    4449 scope.go:122] "RemoveContainer" containerID="d962a00c6aa93312933d4291fdbf5c290952111d56e3b4657c157b35e7af97c7"
Nov 02 09:50:44.716161 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:50:44.716143    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d962a00c6aa93312933d4291fdbf5c290952111d56e3b4657c157b35e7af97c7\": container with ID starting with d962a00c6aa93312933d4291fdbf5c290952111d56e3b4657c157b35e7af97c7 not found: ID does not exist" containerID="d962a00c6aa93312933d4291fdbf5c290952111d56e3b4657c157b35e7af97c7"
Nov 02 09:50:44.716287 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.716169    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d962a00c6aa93312933d4291fdbf5c290952111d56e3b4657c157b35e7af97c7"} err="failed to get container status \"d962a00c6aa93312933d4291fdbf5c290952111d56e3b4657c157b35e7af97c7\": rpc error: code = NotFound desc = could not find container \"d962a00c6aa93312933d4291fdbf5c290952111d56e3b4657c157b35e7af97c7\": container with ID starting with d962a00c6aa93312933d4291fdbf5c290952111d56e3b4657c157b35e7af97c7 not found: ID does not exist"
Nov 02 09:50:44.722838 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.722795    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"]
Nov 02 09:50:44.730379 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.730343    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-27nd6"]
Nov 02 09:50:44.777734 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.777698    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb8wd\" (UniqueName: \"kubernetes.io/projected/981a9928-8e77-4f76-96a1-59f1e541c65e-kube-api-access-nb8wd\") pod \"maas-api-revoke-sub-bf0004c2-kn2lx\" (UID: \"981a9928-8e77-4f76-96a1-59f1e541c65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"
Nov 02 09:50:44.777901 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.777855    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0004c2-kn2lx\" (UID: \"981a9928-8e77-4f76-96a1-59f1e541c65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"
Nov 02 09:50:44.777946 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.777917    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0004c2-kn2lx\" (UID: \"981a9928-8e77-4f76-96a1-59f1e541c65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"
Nov 02 09:50:44.778750 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.778728    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0004c2-kn2lx\" (UID: \"981a9928-8e77-4f76-96a1-59f1e541c65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"
Nov 02 09:50:44.780641 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.780610    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0004c2-kn2lx\" (UID: \"981a9928-8e77-4f76-96a1-59f1e541c65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"
Nov 02 09:50:44.790714 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.790681    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb8wd\" (UniqueName: \"kubernetes.io/projected/981a9928-8e77-4f76-96a1-59f1e541c65e-kube-api-access-nb8wd\") pod \"maas-api-revoke-sub-bf0004c2-kn2lx\" (UID: \"981a9928-8e77-4f76-96a1-59f1e541c65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"
Nov 02 09:50:44.838751 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.838726    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"
Nov 02 09:50:44.878772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.878735    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-xks-ca\") pod \"25edb50b-6bac-4c2d-9045-f1be89706a0b\" (UID: \"25edb50b-6bac-4c2d-9045-f1be89706a0b\") "
Nov 02 09:50:44.878937 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.878798    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-openshift-ca\") pod \"25edb50b-6bac-4c2d-9045-f1be89706a0b\" (UID: \"25edb50b-6bac-4c2d-9045-f1be89706a0b\") "
Nov 02 09:50:44.878937 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.878913    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25edb50b-6bac-4c2d-9045-f1be89706a0b-kube-api-access-5cwmj\" (UniqueName: \"kubernetes.io/projected/25edb50b-6bac-4c2d-9045-f1be89706a0b-kube-api-access-5cwmj\") pod \"25edb50b-6bac-4c2d-9045-f1be89706a0b\" (UID: \"25edb50b-6bac-4c2d-9045-f1be89706a0b\") "
Nov 02 09:50:44.879148 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.879120    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-openshift-ca" pod "25edb50b-6bac-4c2d-9045-f1be89706a0b" (UID: "25edb50b-6bac-4c2d-9045-f1be89706a0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:44.879597 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.879569    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.881154 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.881122    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-xks-ca" pod "25edb50b-6bac-4c2d-9045-f1be89706a0b" (UID: "25edb50b-6bac-4c2d-9045-f1be89706a0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:50:44.881256 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.881161    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25edb50b-6bac-4c2d-9045-f1be89706a0b-kube-api-access-5cwmj" pod "25edb50b-6bac-4c2d-9045-f1be89706a0b" (UID: "25edb50b-6bac-4c2d-9045-f1be89706a0b"). InnerVolumeSpecName "kube-api-access-5cwmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:44.980685 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.980595    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25edb50b-6bac-4c2d-9045-f1be89706a0b-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.980685 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.980631    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cwmj\" (UniqueName: \"kubernetes.io/projected/25edb50b-6bac-4c2d-9045-f1be89706a0b-kube-api-access-5cwmj\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.990357 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:44.990322    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"
Nov 02 09:50:45.208187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:45.208152    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5928113f-5ce3-4403-b390-d52a356939af" path="/var/lib/kubelet/pods/5928113f-5ce3-4403-b390-d52a356939af/volumes"
Nov 02 09:50:45.281916 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:45.281882    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981a9928_8e77_4f76_96a1_59f1e541c65e.slice/crio-f9adf6280c3a282f5e0eddd7c5a032dce05f0332c7c23c042310494bcf74128a WatchSource:0}: Error finding container f9adf6280c3a282f5e0eddd7c5a032dce05f0332c7c23c042310494bcf74128a: Status 404 returned error can't find the container with id f9adf6280c3a282f5e0eddd7c5a032dce05f0332c7c23c042310494bcf74128a
Nov 02 09:50:45.283006 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:45.282975    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"]
Nov 02 09:50:45.693174 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:45.693131    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx" event={"ID":"981a9928-8e77-4f76-96a1-59f1e541c65e","Type":"ContainerStarted","Data":"8b5bf408ff55115002b4ed8d4ad413bd68f274db61398b07b456fcce8929285a"}
Nov 02 09:50:45.693662 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:45.693186    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx" event={"ID":"981a9928-8e77-4f76-96a1-59f1e541c65e","Type":"ContainerStarted","Data":"f9adf6280c3a282f5e0eddd7c5a032dce05f0332c7c23c042310494bcf74128a"}
Nov 02 09:50:45.694784 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:45.694755    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"
Nov 02 09:50:45.694908 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:45.694784    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm" event={"ID":"25edb50b-6bac-4c2d-9045-f1be89706a0b","Type":"ContainerDied","Data":"c45f6562496e9c5616c0ff16f9c484c051383639c2c2eb3ac1bbb9c263b661f6"}
Nov 02 09:50:45.694908 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:45.694829    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c45f6562496e9c5616c0ff16f9c484c051383639c2c2eb3ac1bbb9c263b661f6"
Nov 02 09:50:45.712380 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:45.712307    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx" podStartSLOduration=1.712291795 podStartE2EDuration="1.712291795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:45.71081248 +0000 UTC m=+922.649099275" watchObservedRunningTime="2026-11-02 09:50:45.712291795 +0000 UTC m=+922.650578570"
Nov 02 09:50:46.705097 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:46.705062    4449 generic.go:356] "Generic (PLEG): container finished" podID="981a9928-8e77-4f76-96a1-59f1e541c65e" containerID="8b5bf408ff55115002b4ed8d4ad413bd68f274db61398b07b456fcce8929285a" exitCode=0
Nov 02 09:50:46.705577 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:46.705110    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx" event={"ID":"981a9928-8e77-4f76-96a1-59f1e541c65e","Type":"ContainerDied","Data":"8b5bf408ff55115002b4ed8d4ad413bd68f274db61398b07b456fcce8929285a"}
Nov 02 09:50:47.868978 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:47.868949    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"
Nov 02 09:50:47.931110 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:47.931068    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-openshift-ca\") pod \"981a9928-8e77-4f76-96a1-59f1e541c65e\" (UID: \"981a9928-8e77-4f76-96a1-59f1e541c65e\") "
Nov 02 09:50:47.931351 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:47.931156    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-xks-ca\") pod \"981a9928-8e77-4f76-96a1-59f1e541c65e\" (UID: \"981a9928-8e77-4f76-96a1-59f1e541c65e\") "
Nov 02 09:50:47.931351 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:47.931339    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981a9928-8e77-4f76-96a1-59f1e541c65e-kube-api-access-nb8wd\" (UniqueName: \"kubernetes.io/projected/981a9928-8e77-4f76-96a1-59f1e541c65e-kube-api-access-nb8wd\") pod \"981a9928-8e77-4f76-96a1-59f1e541c65e\" (UID: \"981a9928-8e77-4f76-96a1-59f1e541c65e\") "
Nov 02 09:50:47.934091 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:47.934055    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-xks-ca" pod "981a9928-8e77-4f76-96a1-59f1e541c65e" (UID: "981a9928-8e77-4f76-96a1-59f1e541c65e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:50:47.934312 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:47.934286    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981a9928-8e77-4f76-96a1-59f1e541c65e-kube-api-access-nb8wd" pod "981a9928-8e77-4f76-96a1-59f1e541c65e" (UID: "981a9928-8e77-4f76-96a1-59f1e541c65e"). InnerVolumeSpecName "kube-api-access-nb8wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:47.943200 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:47.943156    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-openshift-ca" pod "981a9928-8e77-4f76-96a1-59f1e541c65e" (UID: "981a9928-8e77-4f76-96a1-59f1e541c65e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:48.033814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:48.033725    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:48.033814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:48.033764    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981a9928-8e77-4f76-96a1-59f1e541c65e-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:48.033814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:48.033780    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb8wd\" (UniqueName: \"kubernetes.io/projected/981a9928-8e77-4f76-96a1-59f1e541c65e-kube-api-access-nb8wd\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:48.535963 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:48.535936    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:50:48.729899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:48.729868    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"
Nov 02 09:50:48.730069 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:48.729866    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx" event={"ID":"981a9928-8e77-4f76-96a1-59f1e541c65e","Type":"ContainerDied","Data":"f9adf6280c3a282f5e0eddd7c5a032dce05f0332c7c23c042310494bcf74128a"}
Nov 02 09:50:48.730069 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:48.730032    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9adf6280c3a282f5e0eddd7c5a032dce05f0332c7c23c042310494bcf74128a"
Nov 02 09:50:49.602203 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.602172    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"]
Nov 02 09:50:49.602868 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.602852    4449 cpu_manager.go:414] "containerMap: removing container" podUID="981a9928-8e77-4f76-96a1-59f1e541c65e" containerName="revoke-keys"
Nov 02 09:50:49.602929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.602869    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981a9928-8e77-4f76-96a1-59f1e541c65e" containerName="revoke-keys"
Nov 02 09:50:49.602929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.602887    4449 cpu_manager.go:414] "containerMap: removing container" podUID="25edb50b-6bac-4c2d-9045-f1be89706a0b" containerName="revoke-keys"
Nov 02 09:50:49.602929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.602892    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25edb50b-6bac-4c2d-9045-f1be89706a0b" containerName="revoke-keys"
Nov 02 09:50:49.603025 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.602994    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="981a9928-8e77-4f76-96a1-59f1e541c65e" containerName="revoke-keys"
Nov 02 09:50:49.603025 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.603004    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="25edb50b-6bac-4c2d-9045-f1be89706a0b" containerName="revoke-keys"
Nov 02 09:50:49.607176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.607156    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.620313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.620286    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"]
Nov 02 09:50:49.620538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.620465    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-712810ea-openshift-default-dockercfg-zbfsq\""
Nov 02 09:50:49.652287 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.652240    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-socket\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.652486 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.652404    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqlqf\" (UniqueName: \"kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-kube-api-access-wqlqf\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.652620 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.652595    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f0ca6c-1150-4c10-9510-3613a96e425c-istiod-ca-cert\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.652762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.652632    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-certs\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.652762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.652665    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-data\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.652897 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.652788    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-token\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.652897 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.652820    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-credential-socket\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.652897 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.652872    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-podinfo\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.653058 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.653001    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-envoy\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.758847 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.757918    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqlqf\" (UniqueName: \"kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-kube-api-access-wqlqf\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.759230 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.759187    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f0ca6c-1150-4c10-9510-3613a96e425c-istiod-ca-cert\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.759387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.759368    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-certs\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.759487 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.759408    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-data\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.759487 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.759451    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-token\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.759592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.759481    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-credential-socket\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.759592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.759540    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-podinfo\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.759592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.759573    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-envoy\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.759735 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.759666    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-socket\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.760924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.760012    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f0ca6c-1150-4c10-9510-3613a96e425c-istiod-ca-cert\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.760924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.760119    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-socket\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.760924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.760397    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-credential-socket\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.760924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.760565    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-certs\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.760924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.760846    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-data\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.781554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.764257    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-envoy\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.781554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.779387    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-podinfo\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.799582 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.797715    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqlqf\" (UniqueName: \"kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-kube-api-access-wqlqf\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.810684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.810589    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-token\") pod \"e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") " pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:49.928250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:49.928187    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:50.141830 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.141803    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"
Nov 02 09:50:50.221099 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:50.221069    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f0ca6c_1150_4c10_9510_3613a96e425c.slice/crio-8e63b4eb4b083f60fd73d6b84686a561d737701462f79255763b676bcb66c46c WatchSource:0}: Error finding container 8e63b4eb4b083f60fd73d6b84686a561d737701462f79255763b676bcb66c46c: Status 404 returned error can't find the container with id 8e63b4eb4b083f60fd73d6b84686a561d737701462f79255763b676bcb66c46c
Nov 02 09:50:50.221599 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.221581    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"]
Nov 02 09:50:50.223501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.223463    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:50:50.223594 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.223531    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:50:50.223594 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.223557    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:50:50.266558 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.266529    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4024dd80-c4b0-4a24-a983-ba34eb385c57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4024dd80-c4b0-4a24-a983-ba34eb385c57-plugins-config-volume\") pod \"4024dd80-c4b0-4a24-a983-ba34eb385c57\" (UID: \"4024dd80-c4b0-4a24-a983-ba34eb385c57\") "
Nov 02 09:50:50.266731 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.266715    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4024dd80-c4b0-4a24-a983-ba34eb385c57-kube-api-access-c9m7z\" (UniqueName: \"kubernetes.io/projected/4024dd80-c4b0-4a24-a983-ba34eb385c57-kube-api-access-c9m7z\") pod \"4024dd80-c4b0-4a24-a983-ba34eb385c57\" (UID: \"4024dd80-c4b0-4a24-a983-ba34eb385c57\") "
Nov 02 09:50:50.266981 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.266946    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4024dd80-c4b0-4a24-a983-ba34eb385c57-plugins-config-volume" pod "4024dd80-c4b0-4a24-a983-ba34eb385c57" (UID: "4024dd80-c4b0-4a24-a983-ba34eb385c57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:50.267333 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.267312    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4024dd80-c4b0-4a24-a983-ba34eb385c57-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:50.269679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.269651    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4024dd80-c4b0-4a24-a983-ba34eb385c57-kube-api-access-c9m7z" pod "4024dd80-c4b0-4a24-a983-ba34eb385c57" (UID: "4024dd80-c4b0-4a24-a983-ba34eb385c57"). InnerVolumeSpecName "kube-api-access-c9m7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:50.368483 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.368440    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9m7z\" (UniqueName: \"kubernetes.io/projected/4024dd80-c4b0-4a24-a983-ba34eb385c57-kube-api-access-c9m7z\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:50.831800 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.831763    4449 generic.go:356] "Generic (PLEG): container finished" podID="4024dd80-c4b0-4a24-a983-ba34eb385c57" containerID="5df023dce23c1f31fac6da011c97f09adcfc20dcb72fa5567a3994efcea4e436" exitCode=0
Nov 02 09:50:50.832265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.831820    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"
Nov 02 09:50:50.832265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.831859    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d" event={"ID":"4024dd80-c4b0-4a24-a983-ba34eb385c57","Type":"ContainerDied","Data":"5df023dce23c1f31fac6da011c97f09adcfc20dcb72fa5567a3994efcea4e436"}
Nov 02 09:50:50.832265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.831910    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d" event={"ID":"4024dd80-c4b0-4a24-a983-ba34eb385c57","Type":"ContainerDied","Data":"7db96204689d1732123823c935a43bda06da337c094e7737e05bf05980371862"}
Nov 02 09:50:50.832265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.831945    4449 scope.go:122] "RemoveContainer" containerID="5df023dce23c1f31fac6da011c97f09adcfc20dcb72fa5567a3994efcea4e436"
Nov 02 09:50:50.834038 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.834012    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" event={"ID":"22f0ca6c-1150-4c10-9510-3613a96e425c","Type":"ContainerStarted","Data":"2f27071bfe5e8042abf5139c2229932e2326959715a89dec6a852cc70040987a"}
Nov 02 09:50:50.834167 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.834054    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" event={"ID":"22f0ca6c-1150-4c10-9510-3613a96e425c","Type":"ContainerStarted","Data":"8e63b4eb4b083f60fd73d6b84686a561d737701462f79255763b676bcb66c46c"}
Nov 02 09:50:50.845624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.845606    4449 scope.go:122] "RemoveContainer" containerID="5df023dce23c1f31fac6da011c97f09adcfc20dcb72fa5567a3994efcea4e436"
Nov 02 09:50:50.845915 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:50:50.845894    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df023dce23c1f31fac6da011c97f09adcfc20dcb72fa5567a3994efcea4e436\": container with ID starting with 5df023dce23c1f31fac6da011c97f09adcfc20dcb72fa5567a3994efcea4e436 not found: ID does not exist" containerID="5df023dce23c1f31fac6da011c97f09adcfc20dcb72fa5567a3994efcea4e436"
Nov 02 09:50:50.845988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.845925    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df023dce23c1f31fac6da011c97f09adcfc20dcb72fa5567a3994efcea4e436"} err="failed to get container status \"5df023dce23c1f31fac6da011c97f09adcfc20dcb72fa5567a3994efcea4e436\": rpc error: code = NotFound desc = could not find container \"5df023dce23c1f31fac6da011c97f09adcfc20dcb72fa5567a3994efcea4e436\": container with ID starting with 5df023dce23c1f31fac6da011c97f09adcfc20dcb72fa5567a3994efcea4e436 not found: ID does not exist"
Nov 02 09:50:50.862266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.862229    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" podStartSLOduration=1.862200814 podStartE2EDuration="1.862200814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:50.858061793 +0000 UTC m=+927.796348585" watchObservedRunningTime="2026-11-02 09:50:50.862200814 +0000 UTC m=+927.800487590"
Nov 02 09:50:50.875013 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.874992    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"]
Nov 02 09:50:50.879621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.879600    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8743a5-6f4c48bc79-lb58d"]
Nov 02 09:50:50.928527 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:50.928496    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:51.209428 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:51.209395    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4024dd80-c4b0-4a24-a983-ba34eb385c57" path="/var/lib/kubelet/pods/4024dd80-c4b0-4a24-a983-ba34eb385c57/volumes"
Nov 02 09:50:52.023391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:52.023330    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:52.967636 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:52.967567    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:52.969251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:52.969084    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:50:54.264267 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.264207    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"]
Nov 02 09:50:54.266989 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.266947    4449 cpu_manager.go:414] "containerMap: removing container" podUID="4024dd80-c4b0-4a24-a983-ba34eb385c57" containerName="payload-processing"
Nov 02 09:50:54.267181 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.267163    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4024dd80-c4b0-4a24-a983-ba34eb385c57" containerName="payload-processing"
Nov 02 09:50:54.267653 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.267631    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="4024dd80-c4b0-4a24-a983-ba34eb385c57" containerName="payload-processing"
Nov 02 09:50:54.294966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.294933    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"]
Nov 02 09:50:54.295154 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.295054    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"
Nov 02 09:50:54.299322 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.299303    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-712810ea\""
Nov 02 09:50:54.300795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.300771    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"]
Nov 02 09:50:54.300911 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.300810    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"]
Nov 02 09:50:54.300911 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.300874    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"
Nov 02 09:50:54.305568 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.305520    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-712810ea-dockercfg-4gmgk\""
Nov 02 09:50:54.479879 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.479841    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx698\" (UniqueName: \"kubernetes.io/projected/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-kube-api-access-kx698\") pod \"payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv\" (UID: \"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"
Nov 02 09:50:54.480083 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.480036    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv\" (UID: \"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"
Nov 02 09:50:54.480193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.480179    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616acf36-b808-4182-bfc6-55ec2849bc0c-plugins-config-volume\") pod \"payload-processing-e2e-ait-712810ea-8449cd864f-l84rs\" (UID: \"616acf36-b808-4182-bfc6-55ec2849bc0c\") " pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"
Nov 02 09:50:54.480279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.480206    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-984xn\" (UniqueName: \"kubernetes.io/projected/616acf36-b808-4182-bfc6-55ec2849bc0c-kube-api-access-984xn\") pod \"payload-processing-e2e-ait-712810ea-8449cd864f-l84rs\" (UID: \"616acf36-b808-4182-bfc6-55ec2849bc0c\") " pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"
Nov 02 09:50:54.581734 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.581696    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx698\" (UniqueName: \"kubernetes.io/projected/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-kube-api-access-kx698\") pod \"payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv\" (UID: \"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"
Nov 02 09:50:54.581930 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.581814    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv\" (UID: \"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"
Nov 02 09:50:54.581930 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.581892    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616acf36-b808-4182-bfc6-55ec2849bc0c-plugins-config-volume\") pod \"payload-processing-e2e-ait-712810ea-8449cd864f-l84rs\" (UID: \"616acf36-b808-4182-bfc6-55ec2849bc0c\") " pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"
Nov 02 09:50:54.581930 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.581915    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984xn\" (UniqueName: \"kubernetes.io/projected/616acf36-b808-4182-bfc6-55ec2849bc0c-kube-api-access-984xn\") pod \"payload-processing-e2e-ait-712810ea-8449cd864f-l84rs\" (UID: \"616acf36-b808-4182-bfc6-55ec2849bc0c\") " pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"
Nov 02 09:50:54.582637 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.582613    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv\" (UID: \"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"
Nov 02 09:50:54.582733 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.582666    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616acf36-b808-4182-bfc6-55ec2849bc0c-plugins-config-volume\") pod \"payload-processing-e2e-ait-712810ea-8449cd864f-l84rs\" (UID: \"616acf36-b808-4182-bfc6-55ec2849bc0c\") " pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"
Nov 02 09:50:54.594608 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.594507    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-984xn\" (UniqueName: \"kubernetes.io/projected/616acf36-b808-4182-bfc6-55ec2849bc0c-kube-api-access-984xn\") pod \"payload-processing-e2e-ait-712810ea-8449cd864f-l84rs\" (UID: \"616acf36-b808-4182-bfc6-55ec2849bc0c\") " pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"
Nov 02 09:50:54.594608 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.594540    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx698\" (UniqueName: \"kubernetes.io/projected/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-kube-api-access-kx698\") pod \"payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv\" (UID: \"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"
Nov 02 09:50:54.618900 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.618871    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"
Nov 02 09:50:54.626536 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.626441    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"
Nov 02 09:50:54.879332 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.879246    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"]
Nov 02 09:50:54.885479 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.885453    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"
Nov 02 09:50:54.900381 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.900269    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:50:54.900381 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.900305    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:50:54.900381 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.900341    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:50:54.901469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.901419    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:50:54.902155 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.902095    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"]
Nov 02 09:50:54.932648 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:54.932606    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616acf36_b808_4182_bfc6_55ec2849bc0c.slice/crio-510c4c185eabc452d007c055eb8a499a2e7689b86ed8934657a853c3f17389e8 WatchSource:0}: Error finding container 510c4c185eabc452d007c055eb8a499a2e7689b86ed8934657a853c3f17389e8: Status 404 returned error can't find the container with id 510c4c185eabc452d007c055eb8a499a2e7689b86ed8934657a853c3f17389e8
Nov 02 09:50:54.932912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.932886    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"]
Nov 02 09:50:54.933463 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:54.933437    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e1cfb2_c0f2_40e1_b1eb_22f2c14b2aee.slice/crio-5f90ee368e694f58d4828c3036983aaf2d8cfb28a2c0344ec814b3e3948b433d WatchSource:0}: Error finding container 5f90ee368e694f58d4828c3036983aaf2d8cfb28a2c0344ec814b3e3948b433d: Status 404 returned error can't find the container with id 5f90ee368e694f58d4828c3036983aaf2d8cfb28a2c0344ec814b3e3948b433d
Nov 02 09:50:54.939368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.939346    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"]
Nov 02 09:50:54.990784 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.990755    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6kxn\" (UniqueName: \"kubernetes.io/projected/da0c263d-3489-4d88-b151-7746d205493e-kube-api-access-m6kxn\") pod \"maas-api-revoke-sub-d66aab2e-m8zcz\" (UID: \"da0c263d-3489-4d88-b151-7746d205493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"
Nov 02 09:50:54.990907 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.990808    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0c263d-3489-4d88-b151-7746d205493e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66aab2e-m8zcz\" (UID: \"da0c263d-3489-4d88-b151-7746d205493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"
Nov 02 09:50:54.990971 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:54.990912    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0c263d-3489-4d88-b151-7746d205493e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66aab2e-m8zcz\" (UID: \"da0c263d-3489-4d88-b151-7746d205493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"
Nov 02 09:50:55.092405 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.092365    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6kxn\" (UniqueName: \"kubernetes.io/projected/da0c263d-3489-4d88-b151-7746d205493e-kube-api-access-m6kxn\") pod \"maas-api-revoke-sub-d66aab2e-m8zcz\" (UID: \"da0c263d-3489-4d88-b151-7746d205493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"
Nov 02 09:50:55.092617 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.092414    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0c263d-3489-4d88-b151-7746d205493e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66aab2e-m8zcz\" (UID: \"da0c263d-3489-4d88-b151-7746d205493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"
Nov 02 09:50:55.092617 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.092478    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0c263d-3489-4d88-b151-7746d205493e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66aab2e-m8zcz\" (UID: \"da0c263d-3489-4d88-b151-7746d205493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"
Nov 02 09:50:55.093114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.093091    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0c263d-3489-4d88-b151-7746d205493e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66aab2e-m8zcz\" (UID: \"da0c263d-3489-4d88-b151-7746d205493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"
Nov 02 09:50:55.095050 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.095025    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0c263d-3489-4d88-b151-7746d205493e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66aab2e-m8zcz\" (UID: \"da0c263d-3489-4d88-b151-7746d205493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"
Nov 02 09:50:55.107123 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.107097    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6kxn\" (UniqueName: \"kubernetes.io/projected/da0c263d-3489-4d88-b151-7746d205493e-kube-api-access-m6kxn\") pod \"maas-api-revoke-sub-d66aab2e-m8zcz\" (UID: \"da0c263d-3489-4d88-b151-7746d205493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"
Nov 02 09:50:55.225686 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.225592    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"
Nov 02 09:50:55.299686 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.299643    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv" event={"ID":"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee","Type":"ContainerStarted","Data":"261902a77bcfe3f33718220b4a90113fb2ca393b23af483e28033eddd6e7a31d"}
Nov 02 09:50:55.300161 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.299718    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv" event={"ID":"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee","Type":"ContainerStarted","Data":"5f90ee368e694f58d4828c3036983aaf2d8cfb28a2c0344ec814b3e3948b433d"}
Nov 02 09:50:55.300819 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.300794    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"
Nov 02 09:50:55.303483 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.303448    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs" event={"ID":"616acf36-b808-4182-bfc6-55ec2849bc0c","Type":"ContainerStarted","Data":"72b277a2fa5ba2fe67b54fc4ad13cc741419ff02cedb6d560e91c9f9620e6c06"}
Nov 02 09:50:55.303601 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.303541    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs" event={"ID":"616acf36-b808-4182-bfc6-55ec2849bc0c","Type":"ContainerStarted","Data":"510c4c185eabc452d007c055eb8a499a2e7689b86ed8934657a853c3f17389e8"}
Nov 02 09:50:55.305101 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.304321    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"
Nov 02 09:50:55.323852 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.323799    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv" podStartSLOduration=2.323782468 podStartE2EDuration="2.323782468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:55.320493571 +0000 UTC m=+932.258780350" watchObservedRunningTime="2026-11-02 09:50:55.323782468 +0000 UTC m=+932.262069243"
Nov 02 09:50:55.350657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.350588    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs" podStartSLOduration=2.350566052 podStartE2EDuration="2.350566052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:55.344578083 +0000 UTC m=+932.282864861" watchObservedRunningTime="2026-11-02 09:50:55.350566052 +0000 UTC m=+932.288852829"
Nov 02 09:50:55.512130 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:55.512097    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0c263d_3489_4d88_b151_7746d205493e.slice/crio-8e22dc08f365af5c1aa6ede42caf4f49854dca90d8ae08de7c8ab0c6a44407c9 WatchSource:0}: Error finding container 8e22dc08f365af5c1aa6ede42caf4f49854dca90d8ae08de7c8ab0c6a44407c9: Status 404 returned error can't find the container with id 8e22dc08f365af5c1aa6ede42caf4f49854dca90d8ae08de7c8ab0c6a44407c9
Nov 02 09:50:55.512818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:55.512793    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"]
Nov 02 09:50:56.315211 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:56.315174    4449 generic.go:356] "Generic (PLEG): container finished" podID="da0c263d-3489-4d88-b151-7746d205493e" containerID="daac8a526e8b72699d57fcdde310babc1313661f981a857067494c545604c4ef" exitCode=0
Nov 02 09:50:56.315705 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:56.315261    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz" event={"ID":"da0c263d-3489-4d88-b151-7746d205493e","Type":"ContainerDied","Data":"daac8a526e8b72699d57fcdde310babc1313661f981a857067494c545604c4ef"}
Nov 02 09:50:56.315705 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:56.315328    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz" event={"ID":"da0c263d-3489-4d88-b151-7746d205493e","Type":"ContainerStarted","Data":"8e22dc08f365af5c1aa6ede42caf4f49854dca90d8ae08de7c8ab0c6a44407c9"}
Nov 02 09:50:57.489236 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.489189    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"
Nov 02 09:50:57.525196 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.525157    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0c263d-3489-4d88-b151-7746d205493e-kube-api-access-m6kxn\" (UniqueName: \"kubernetes.io/projected/da0c263d-3489-4d88-b151-7746d205493e-kube-api-access-m6kxn\") pod \"da0c263d-3489-4d88-b151-7746d205493e\" (UID: \"da0c263d-3489-4d88-b151-7746d205493e\") "
Nov 02 09:50:57.525420 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.525399    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da0c263d-3489-4d88-b151-7746d205493e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0c263d-3489-4d88-b151-7746d205493e-maas-api-xks-ca\") pod \"da0c263d-3489-4d88-b151-7746d205493e\" (UID: \"da0c263d-3489-4d88-b151-7746d205493e\") "
Nov 02 09:50:57.525516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.525481    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0c263d-3489-4d88-b151-7746d205493e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0c263d-3489-4d88-b151-7746d205493e-maas-api-openshift-ca\") pod \"da0c263d-3489-4d88-b151-7746d205493e\" (UID: \"da0c263d-3489-4d88-b151-7746d205493e\") "
Nov 02 09:50:57.525902 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.525868    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0c263d-3489-4d88-b151-7746d205493e-maas-api-openshift-ca" pod "da0c263d-3489-4d88-b151-7746d205493e" (UID: "da0c263d-3489-4d88-b151-7746d205493e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:57.526288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.526265    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0c263d-3489-4d88-b151-7746d205493e-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:57.527685 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.527658    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0c263d-3489-4d88-b151-7746d205493e-kube-api-access-m6kxn" pod "da0c263d-3489-4d88-b151-7746d205493e" (UID: "da0c263d-3489-4d88-b151-7746d205493e"). InnerVolumeSpecName "kube-api-access-m6kxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:57.528203 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.528178    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da0c263d-3489-4d88-b151-7746d205493e-maas-api-xks-ca" pod "da0c263d-3489-4d88-b151-7746d205493e" (UID: "da0c263d-3489-4d88-b151-7746d205493e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:50:57.628148 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.628040    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0c263d-3489-4d88-b151-7746d205493e-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:57.628148 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.628077    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6kxn\" (UniqueName: \"kubernetes.io/projected/da0c263d-3489-4d88-b151-7746d205493e-kube-api-access-m6kxn\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:57.884134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.884054    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"]
Nov 02 09:50:57.884759 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.884744    4449 cpu_manager.go:414] "containerMap: removing container" podUID="da0c263d-3489-4d88-b151-7746d205493e" containerName="revoke-keys"
Nov 02 09:50:57.884801 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.884760    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0c263d-3489-4d88-b151-7746d205493e" containerName="revoke-keys"
Nov 02 09:50:57.884861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.884852    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0c263d-3489-4d88-b151-7746d205493e" containerName="revoke-keys"
Nov 02 09:50:57.888672 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.888654    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"
Nov 02 09:50:57.896640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.896610    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"]
Nov 02 09:50:57.932026 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.931983    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q499p\" (UniqueName: \"kubernetes.io/projected/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-kube-api-access-q499p\") pod \"maas-api-revoke-sub-3d493b49-c8fh4\" (UID: \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"
Nov 02 09:50:57.932205 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.932157    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d493b49-c8fh4\" (UID: \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"
Nov 02 09:50:57.932461 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:57.932437    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d493b49-c8fh4\" (UID: \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"
Nov 02 09:50:58.034262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.034209    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q499p\" (UniqueName: \"kubernetes.io/projected/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-kube-api-access-q499p\") pod \"maas-api-revoke-sub-3d493b49-c8fh4\" (UID: \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"
Nov 02 09:50:58.034472 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.034295    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d493b49-c8fh4\" (UID: \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"
Nov 02 09:50:58.034472 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.034431    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d493b49-c8fh4\" (UID: \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"
Nov 02 09:50:58.034941 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.034919    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d493b49-c8fh4\" (UID: \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"
Nov 02 09:50:58.037039 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.037015    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d493b49-c8fh4\" (UID: \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"
Nov 02 09:50:58.048452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.048413    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q499p\" (UniqueName: \"kubernetes.io/projected/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-kube-api-access-q499p\") pod \"maas-api-revoke-sub-3d493b49-c8fh4\" (UID: \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"
Nov 02 09:50:58.209405 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.209317    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"
Nov 02 09:50:58.238933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.238891    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"]
Nov 02 09:50:58.248388 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.246532    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.251977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.251948    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"]
Nov 02 09:50:58.342270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.338545    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"
Nov 02 09:50:58.342270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.338574    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz" event={"ID":"da0c263d-3489-4d88-b151-7746d205493e","Type":"ContainerDied","Data":"8e22dc08f365af5c1aa6ede42caf4f49854dca90d8ae08de7c8ab0c6a44407c9"}
Nov 02 09:50:58.342270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.338627    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e22dc08f365af5c1aa6ede42caf4f49854dca90d8ae08de7c8ab0c6a44407c9"
Nov 02 09:50:58.347810 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.347622    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25wj9\" (UniqueName: \"kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-kube-api-access-25wj9\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.347810 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.347734    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.347957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.347818    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.347957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.347861    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.449236 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.449179    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25wj9\" (UniqueName: \"kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-kube-api-access-25wj9\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.449437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.449306    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.449437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.449399    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.449547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.449442    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.450228 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.450190    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.450327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.450201    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.452320 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.452284    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.461922 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.461856    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25wj9\" (UniqueName: \"kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-kube-api-access-25wj9\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.541066 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:58.541030    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a15304e_8f78_4fda_a6dc_8f2cd9a22afb.slice/crio-36cc3d706f745ccdbe9e7d6709c1e668db660da825f670de086fea5e56b6d079 WatchSource:0}: Error finding container 36cc3d706f745ccdbe9e7d6709c1e668db660da825f670de086fea5e56b6d079: Status 404 returned error can't find the container with id 36cc3d706f745ccdbe9e7d6709c1e668db660da825f670de086fea5e56b6d079
Nov 02 09:50:58.543528 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.543499    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"]
Nov 02 09:50:58.569175 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.569149    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:58.857998 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:50:58.857970    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34535c8c_9a37_4955_ab9e_da3d09bf0d2d.slice/crio-c9416ded70f123c1e3b67d239c9be8703a51d8f15364692c2015f7978b8af2fc WatchSource:0}: Error finding container c9416ded70f123c1e3b67d239c9be8703a51d8f15364692c2015f7978b8af2fc: Status 404 returned error can't find the container with id c9416ded70f123c1e3b67d239c9be8703a51d8f15364692c2015f7978b8af2fc
Nov 02 09:50:58.858697 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:58.858673    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"]
Nov 02 09:50:59.348814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:59.348775    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4" event={"ID":"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb","Type":"ContainerStarted","Data":"fcdf85c18a5a5f8884323eb3819deed0f3b200c91c02f54ad611d23993cca080"}
Nov 02 09:50:59.349015 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:59.348842    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4" event={"ID":"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb","Type":"ContainerStarted","Data":"36cc3d706f745ccdbe9e7d6709c1e668db660da825f670de086fea5e56b6d079"}
Nov 02 09:50:59.351004 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:59.350967    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g" event={"ID":"34535c8c-9a37-4955-ab9e-da3d09bf0d2d","Type":"ContainerStarted","Data":"48f3510d90f49622654bf57904da427c6f86488f5c62ae32dbda6d8125e7c3ae"}
Nov 02 09:50:59.351156 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:59.351023    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g" event={"ID":"34535c8c-9a37-4955-ab9e-da3d09bf0d2d","Type":"ContainerStarted","Data":"c9416ded70f123c1e3b67d239c9be8703a51d8f15364692c2015f7978b8af2fc"}
Nov 02 09:50:59.351156 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:59.351043    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:50:59.370253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:59.370180    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4" podStartSLOduration=2.370165805 podStartE2EDuration="2.370165805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:59.368287535 +0000 UTC m=+936.306574302" watchObservedRunningTime="2026-11-02 09:50:59.370165805 +0000 UTC m=+936.308452580"
Nov 02 09:50:59.390923 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:50:59.390865    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g" podStartSLOduration=1.390848946 podStartE2EDuration="1.390848946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:59.387723825 +0000 UTC m=+936.326010614" watchObservedRunningTime="2026-11-02 09:50:59.390848946 +0000 UTC m=+936.329135722"
Nov 02 09:51:00.367009 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:00.366964    4449 generic.go:356] "Generic (PLEG): container finished" podID="0a15304e-8f78-4fda-a6dc-8f2cd9a22afb" containerID="fcdf85c18a5a5f8884323eb3819deed0f3b200c91c02f54ad611d23993cca080" exitCode=0
Nov 02 09:51:00.367419 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:00.367054    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4" event={"ID":"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb","Type":"ContainerDied","Data":"fcdf85c18a5a5f8884323eb3819deed0f3b200c91c02f54ad611d23993cca080"}
Nov 02 09:51:01.775138 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:01.775104    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"
Nov 02 09:51:01.880194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:01.880157    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-xks-ca\") pod \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\" (UID: \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\") "
Nov 02 09:51:01.880372 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:01.880350    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-kube-api-access-q499p\" (UniqueName: \"kubernetes.io/projected/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-kube-api-access-q499p\") pod \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\" (UID: \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\") "
Nov 02 09:51:01.880486 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:01.880469    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-openshift-ca\") pod \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\" (UID: \"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb\") "
Nov 02 09:51:01.880864 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:01.880810    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-openshift-ca" pod "0a15304e-8f78-4fda-a6dc-8f2cd9a22afb" (UID: "0a15304e-8f78-4fda-a6dc-8f2cd9a22afb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:01.882519 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:01.882490    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-xks-ca" pod "0a15304e-8f78-4fda-a6dc-8f2cd9a22afb" (UID: "0a15304e-8f78-4fda-a6dc-8f2cd9a22afb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:51:01.882621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:01.882550    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-kube-api-access-q499p" pod "0a15304e-8f78-4fda-a6dc-8f2cd9a22afb" (UID: "0a15304e-8f78-4fda-a6dc-8f2cd9a22afb"). InnerVolumeSpecName "kube-api-access-q499p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:01.982045 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:01.982005    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:01.982045 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:01.982039    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:01.982045 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:01.982050    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q499p\" (UniqueName: \"kubernetes.io/projected/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb-kube-api-access-q499p\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:02.433624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:02.433597    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"
Nov 02 09:51:02.433782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:02.433644    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4" event={"ID":"0a15304e-8f78-4fda-a6dc-8f2cd9a22afb","Type":"ContainerDied","Data":"36cc3d706f745ccdbe9e7d6709c1e668db660da825f670de086fea5e56b6d079"}
Nov 02 09:51:02.433782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:02.433700    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36cc3d706f745ccdbe9e7d6709c1e668db660da825f670de086fea5e56b6d079"
Nov 02 09:51:06.548603 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:51:06.548490    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:51:07.330521 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:07.330487    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"
Nov 02 09:51:07.330669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:07.330570    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"
Nov 02 09:51:09.051502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.051469    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"]
Nov 02 09:51:09.052107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.052092    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0a15304e-8f78-4fda-a6dc-8f2cd9a22afb" containerName="revoke-keys"
Nov 02 09:51:09.052166 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.052108    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a15304e-8f78-4fda-a6dc-8f2cd9a22afb" containerName="revoke-keys"
Nov 02 09:51:09.052272 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.052261    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a15304e-8f78-4fda-a6dc-8f2cd9a22afb" containerName="revoke-keys"
Nov 02 09:51:09.055899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.055881    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"
Nov 02 09:51:09.060764 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.060665    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:51:09.060764 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.060730    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:51:09.061133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.061114    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:51:09.061309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.061167    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:51:09.066617 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.066600    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"]
Nov 02 09:51:09.085150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.085123    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gb9j\" (UniqueName: \"kubernetes.io/projected/bf1721c3-3b9c-49c3-b032-b2a46efe518a-kube-api-access-4gb9j\") pod \"maas-api-revoke-sub-2c9308a5-hs7ss\" (UID: \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"
Nov 02 09:51:09.085296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.085169    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9308a5-hs7ss\" (UID: \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"
Nov 02 09:51:09.085349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.085291    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9308a5-hs7ss\" (UID: \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"
Nov 02 09:51:09.187045 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.187011    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gb9j\" (UniqueName: \"kubernetes.io/projected/bf1721c3-3b9c-49c3-b032-b2a46efe518a-kube-api-access-4gb9j\") pod \"maas-api-revoke-sub-2c9308a5-hs7ss\" (UID: \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"
Nov 02 09:51:09.187209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.187060    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9308a5-hs7ss\" (UID: \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"
Nov 02 09:51:09.187209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.187107    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9308a5-hs7ss\" (UID: \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"
Nov 02 09:51:09.187955 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.187871    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9308a5-hs7ss\" (UID: \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"
Nov 02 09:51:09.189713 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.189691    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9308a5-hs7ss\" (UID: \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"
Nov 02 09:51:09.199652 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.199627    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gb9j\" (UniqueName: \"kubernetes.io/projected/bf1721c3-3b9c-49c3-b032-b2a46efe518a-kube-api-access-4gb9j\") pod \"maas-api-revoke-sub-2c9308a5-hs7ss\" (UID: \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"
Nov 02 09:51:09.372333 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.372210    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"
Nov 02 09:51:09.662386 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:09.662352    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1721c3_3b9c_49c3_b032_b2a46efe518a.slice/crio-da6ecc327cd484e0109a79dbfba40d3150cb4b72a7a935ae94db1bfd0681ab55 WatchSource:0}: Error finding container da6ecc327cd484e0109a79dbfba40d3150cb4b72a7a935ae94db1bfd0681ab55: Status 404 returned error can't find the container with id da6ecc327cd484e0109a79dbfba40d3150cb4b72a7a935ae94db1bfd0681ab55
Nov 02 09:51:09.677494 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:09.674598    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"]
Nov 02 09:51:10.370875 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:10.370849    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:51:10.431842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:10.431812    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"]
Nov 02 09:51:10.546073 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:10.546041    4449 generic.go:356] "Generic (PLEG): container finished" podID="bf1721c3-3b9c-49c3-b032-b2a46efe518a" containerID="1fa7e14ed036e268a811219a7307f0c2ef55f725ecb1f92df16f64e2e9e825ba" exitCode=0
Nov 02 09:51:10.546275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:10.546119    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss" event={"ID":"bf1721c3-3b9c-49c3-b032-b2a46efe518a","Type":"ContainerDied","Data":"1fa7e14ed036e268a811219a7307f0c2ef55f725ecb1f92df16f64e2e9e825ba"}
Nov 02 09:51:10.546275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:10.546182    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss" event={"ID":"bf1721c3-3b9c-49c3-b032-b2a46efe518a","Type":"ContainerStarted","Data":"da6ecc327cd484e0109a79dbfba40d3150cb4b72a7a935ae94db1bfd0681ab55"}
Nov 02 09:51:11.709399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:11.709369    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"
Nov 02 09:51:11.822444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:11.822350    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1721c3-3b9c-49c3-b032-b2a46efe518a-kube-api-access-4gb9j\" (UniqueName: \"kubernetes.io/projected/bf1721c3-3b9c-49c3-b032-b2a46efe518a-kube-api-access-4gb9j\") pod \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\" (UID: \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\") "
Nov 02 09:51:11.822444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:11.822395    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-openshift-ca\") pod \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\" (UID: \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\") "
Nov 02 09:51:11.822681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:11.822463    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-xks-ca\") pod \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\" (UID: \"bf1721c3-3b9c-49c3-b032-b2a46efe518a\") "
Nov 02 09:51:11.822854 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:11.822827    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-openshift-ca" pod "bf1721c3-3b9c-49c3-b032-b2a46efe518a" (UID: "bf1721c3-3b9c-49c3-b032-b2a46efe518a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:11.823044 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:11.823026    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:11.824700 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:11.824675    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1721c3-3b9c-49c3-b032-b2a46efe518a-kube-api-access-4gb9j" pod "bf1721c3-3b9c-49c3-b032-b2a46efe518a" (UID: "bf1721c3-3b9c-49c3-b032-b2a46efe518a"). InnerVolumeSpecName "kube-api-access-4gb9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:11.824807 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:11.824751    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-xks-ca" pod "bf1721c3-3b9c-49c3-b032-b2a46efe518a" (UID: "bf1721c3-3b9c-49c3-b032-b2a46efe518a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:51:11.924891 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:11.924858    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gb9j\" (UniqueName: \"kubernetes.io/projected/bf1721c3-3b9c-49c3-b032-b2a46efe518a-kube-api-access-4gb9j\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:11.924891 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:11.924891    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1721c3-3b9c-49c3-b032-b2a46efe518a-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:12.566992 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:12.566961    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"
Nov 02 09:51:12.566992 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:12.566985    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss" event={"ID":"bf1721c3-3b9c-49c3-b032-b2a46efe518a","Type":"ContainerDied","Data":"da6ecc327cd484e0109a79dbfba40d3150cb4b72a7a935ae94db1bfd0681ab55"}
Nov 02 09:51:12.567203 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:12.567036    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da6ecc327cd484e0109a79dbfba40d3150cb4b72a7a935ae94db1bfd0681ab55"
Nov 02 09:51:13.869645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:13.869308    4449 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Nov 02 09:51:14.031258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:14.027723    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:14.257509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:14.257410    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:14.587344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:14.587298    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": context deadline exceeded"
Nov 02 09:51:14.640430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:14.640373    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 02 09:51:14.641004 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:14.640966    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded"
Nov 02 09:51:15.023489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:15.023361    4449 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/health\": context deadline exceeded"
Nov 02 09:51:15.028353 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:15.028301    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:15.029058 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:15.029037    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:51:15.268246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:15.265666    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:15.473242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:15.442324    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr" podUID="1bb1e8a4-75d1-4b42-995c-089ab025f1c1" containerName="payload-processing" containerID="cri-o://399a06c4f0bca63a64307bad4ed82c869c2e995fb773040e29fc7ad75f11c761" gracePeriod=25
Nov 02 09:51:15.591669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:15.591636    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:51:15.644805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:15.642706    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:15.644805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:15.643246    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:51:16.066248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:16.041363    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:16.278360 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:16.266954    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:16.572261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:16.571362    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:16.647128 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:16.646378    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:16.710040 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:16.699553    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:17.035645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:17.035614    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:51:17.313417 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:17.313309    4449 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:17.479943 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:17.275383    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:17.588305 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:17.580006    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:17.693634 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:17.693574    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 02 09:51:17.735583 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:17.735550    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:51:17.763387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:17.760240    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:51:17.763387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:17.759919    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded"
Nov 02 09:51:18.019298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:17.994595    4449 generic.go:356] "Generic (PLEG): container finished" podID="1bb1e8a4-75d1-4b42-995c-089ab025f1c1" containerID="399a06c4f0bca63a64307bad4ed82c869c2e995fb773040e29fc7ad75f11c761" exitCode=0
Nov 02 09:51:18.019298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:17.994699    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr" event={"ID":"1bb1e8a4-75d1-4b42-995c-089ab025f1c1","Type":"ContainerDied","Data":"399a06c4f0bca63a64307bad4ed82c869c2e995fb773040e29fc7ad75f11c761"}
Nov 02 09:51:18.048779 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.048745    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:51:18.094418 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.094389    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:51:18.097071 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.097043    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:51:18.102787 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.102738    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:51:18.183858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.183825    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:51:18.299497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.299450    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-plugins-config-volume\") pod \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") "
Nov 02 09:51:18.299670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.299652    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-routing-overlay-volume\") pod \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") "
Nov 02 09:51:18.299746 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.299731    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-provider-credentials\") pod \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") "
Nov 02 09:51:18.299858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.299841    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-kube-api-access-gss79\" (UniqueName: \"kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-kube-api-access-gss79\") pod \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\" (UID: \"1bb1e8a4-75d1-4b42-995c-089ab025f1c1\") "
Nov 02 09:51:18.300005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.299860    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-plugins-config-volume" pod "1bb1e8a4-75d1-4b42-995c-089ab025f1c1" (UID: "1bb1e8a4-75d1-4b42-995c-089ab025f1c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:18.300071 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.300033    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-routing-overlay-volume" pod "1bb1e8a4-75d1-4b42-995c-089ab025f1c1" (UID: "1bb1e8a4-75d1-4b42-995c-089ab025f1c1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:18.300557 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.300531    4449 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-routing-overlay-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:18.300684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.300560    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:18.301999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.301976    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-provider-credentials" pod "1bb1e8a4-75d1-4b42-995c-089ab025f1c1" (UID: "1bb1e8a4-75d1-4b42-995c-089ab025f1c1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:18.302468 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.302449    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-kube-api-access-gss79" pod "1bb1e8a4-75d1-4b42-995c-089ab025f1c1" (UID: "1bb1e8a4-75d1-4b42-995c-089ab025f1c1"). InnerVolumeSpecName "kube-api-access-gss79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:18.401834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.401768    4449 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-provider-credentials\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:18.401834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.401809    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gss79\" (UniqueName: \"kubernetes.io/projected/1bb1e8a4-75d1-4b42-995c-089ab025f1c1-kube-api-access-gss79\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:18.463246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.462674    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"]
Nov 02 09:51:18.481342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.476525    4449 cpu_manager.go:414] "containerMap: removing container" podUID="1bb1e8a4-75d1-4b42-995c-089ab025f1c1" containerName="payload-processing"
Nov 02 09:51:18.481342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.476557    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb1e8a4-75d1-4b42-995c-089ab025f1c1" containerName="payload-processing"
Nov 02 09:51:18.481342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.476586    4449 cpu_manager.go:414] "containerMap: removing container" podUID="bf1721c3-3b9c-49c3-b032-b2a46efe518a" containerName="revoke-keys"
Nov 02 09:51:18.481342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.476595    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1721c3-3b9c-49c3-b032-b2a46efe518a" containerName="revoke-keys"
Nov 02 09:51:18.481342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.476880    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb1e8a4-75d1-4b42-995c-089ab025f1c1" containerName="payload-processing"
Nov 02 09:51:18.481342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.476898    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1721c3-3b9c-49c3-b032-b2a46efe518a" containerName="revoke-keys"
Nov 02 09:51:18.487888 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.484602    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"
Nov 02 09:51:18.489247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.489150    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:51:18.489616 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.489553    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:51:18.489763 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.489742    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:51:18.489851 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.489804    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:51:18.490979 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.490954    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"]
Nov 02 09:51:18.503512 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.503483    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33be085e-851a-46a2-b44c-524abb75390a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-712810ea-sftp7\" (UID: \"33be085e-851a-46a2-b44c-524abb75390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"
Nov 02 09:51:18.503661 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.503645    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl652\" (UniqueName: \"kubernetes.io/projected/33be085e-851a-46a2-b44c-524abb75390a-kube-api-access-cl652\") pod \"maas-api-revoke-keys-e2e-ait-712810ea-sftp7\" (UID: \"33be085e-851a-46a2-b44c-524abb75390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"
Nov 02 09:51:18.503732 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.503683    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33be085e-851a-46a2-b44c-524abb75390a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-712810ea-sftp7\" (UID: \"33be085e-851a-46a2-b44c-524abb75390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"
Nov 02 09:51:18.605549 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.605463    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33be085e-851a-46a2-b44c-524abb75390a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-712810ea-sftp7\" (UID: \"33be085e-851a-46a2-b44c-524abb75390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"
Nov 02 09:51:18.605728 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.605608    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl652\" (UniqueName: \"kubernetes.io/projected/33be085e-851a-46a2-b44c-524abb75390a-kube-api-access-cl652\") pod \"maas-api-revoke-keys-e2e-ait-712810ea-sftp7\" (UID: \"33be085e-851a-46a2-b44c-524abb75390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"
Nov 02 09:51:18.605728 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.605656    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33be085e-851a-46a2-b44c-524abb75390a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-712810ea-sftp7\" (UID: \"33be085e-851a-46a2-b44c-524abb75390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"
Nov 02 09:51:18.606122 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.606099    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33be085e-851a-46a2-b44c-524abb75390a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-712810ea-sftp7\" (UID: \"33be085e-851a-46a2-b44c-524abb75390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"
Nov 02 09:51:18.608146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.608107    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33be085e-851a-46a2-b44c-524abb75390a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-712810ea-sftp7\" (UID: \"33be085e-851a-46a2-b44c-524abb75390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"
Nov 02 09:51:18.622289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.622261    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl652\" (UniqueName: \"kubernetes.io/projected/33be085e-851a-46a2-b44c-524abb75390a-kube-api-access-cl652\") pod \"maas-api-revoke-keys-e2e-ait-712810ea-sftp7\" (UID: \"33be085e-851a-46a2-b44c-524abb75390a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"
Nov 02 09:51:18.807786 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.807745    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"
Nov 02 09:51:19.002290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:18.997139    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"]
Nov 02 09:51:19.022469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.022015    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"]
Nov 02 09:51:19.022469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.022128    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.023746 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.023720    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"
Nov 02 09:51:19.026680 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.023798    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr" event={"ID":"1bb1e8a4-75d1-4b42-995c-089ab025f1c1","Type":"ContainerDied","Data":"5a0503958e0ae1321eefad63ff10dfc007c041d76f33da23fc6c4afa8243898f"}
Nov 02 09:51:19.032825 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.032787    4449 scope.go:122] "RemoveContainer" containerID="399a06c4f0bca63a64307bad4ed82c869c2e995fb773040e29fc7ad75f11c761"
Nov 02 09:51:19.115073 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.115034    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8\" (UID: \"81d92016-a072-4478-9231-12996763776e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.115276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.115255    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8\" (UID: \"81d92016-a072-4478-9231-12996763776e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.115453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.115434    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8\" (UID: \"81d92016-a072-4478-9231-12996763776e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.115493 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.115467    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d27p\" (UniqueName: \"kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-kube-api-access-4d27p\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8\" (UID: \"81d92016-a072-4478-9231-12996763776e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.116618 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.116293    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"]
Nov 02 09:51:19.122293 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.122267    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-6c784gzjr"]
Nov 02 09:51:19.208391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.208347    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb1e8a4-75d1-4b42-995c-089ab025f1c1" path="/var/lib/kubelet/pods/1bb1e8a4-75d1-4b42-995c-089ab025f1c1/volumes"
Nov 02 09:51:19.217200 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.217173    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8\" (UID: \"81d92016-a072-4478-9231-12996763776e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.217379 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.217312    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8\" (UID: \"81d92016-a072-4478-9231-12996763776e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.217601 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.217582    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8\" (UID: \"81d92016-a072-4478-9231-12996763776e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.217701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.217609    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d27p\" (UniqueName: \"kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-kube-api-access-4d27p\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8\" (UID: \"81d92016-a072-4478-9231-12996763776e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.218277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.218211    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8\" (UID: \"81d92016-a072-4478-9231-12996763776e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.218667 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.218647    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8\" (UID: \"81d92016-a072-4478-9231-12996763776e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.220282 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.220261    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8\" (UID: \"81d92016-a072-4478-9231-12996763776e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.232356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.232331    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d27p\" (UniqueName: \"kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-kube-api-access-4d27p\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8\" (UID: \"81d92016-a072-4478-9231-12996763776e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.277995 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:19.277958    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33be085e_851a_46a2_b44c_524abb75390a.slice/crio-63d4029a87ea297a48f42e811d2a9f9d75c7c6933a818adec1ac303001f26c89 WatchSource:0}: Error finding container 63d4029a87ea297a48f42e811d2a9f9d75c7c6933a818adec1ac303001f26c89: Status 404 returned error can't find the container with id 63d4029a87ea297a48f42e811d2a9f9d75c7c6933a818adec1ac303001f26c89
Nov 02 09:51:19.280295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.280270    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"]
Nov 02 09:51:19.375999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.375969    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:19.382795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.382766    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"]
Nov 02 09:51:19.387896 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.387457    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"
Nov 02 09:51:19.395682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.395650    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"]
Nov 02 09:51:19.420986 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.420945    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f9de60-s94gj\" (UID: \"0e8be54d-ca5b-4794-b516-7b14c36bc840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"
Nov 02 09:51:19.421500 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.421080    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f9de60-s94gj\" (UID: \"0e8be54d-ca5b-4794-b516-7b14c36bc840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"
Nov 02 09:51:19.421500 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.421336    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhm2q\" (UniqueName: \"kubernetes.io/projected/0e8be54d-ca5b-4794-b516-7b14c36bc840-kube-api-access-xhm2q\") pod \"maas-api-revoke-sub-83f9de60-s94gj\" (UID: \"0e8be54d-ca5b-4794-b516-7b14c36bc840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"
Nov 02 09:51:19.523087 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.523047    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f9de60-s94gj\" (UID: \"0e8be54d-ca5b-4794-b516-7b14c36bc840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"
Nov 02 09:51:19.523294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.523178    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhm2q\" (UniqueName: \"kubernetes.io/projected/0e8be54d-ca5b-4794-b516-7b14c36bc840-kube-api-access-xhm2q\") pod \"maas-api-revoke-sub-83f9de60-s94gj\" (UID: \"0e8be54d-ca5b-4794-b516-7b14c36bc840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"
Nov 02 09:51:19.523294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.523271    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f9de60-s94gj\" (UID: \"0e8be54d-ca5b-4794-b516-7b14c36bc840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"
Nov 02 09:51:19.523858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.523835    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f9de60-s94gj\" (UID: \"0e8be54d-ca5b-4794-b516-7b14c36bc840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"
Nov 02 09:51:19.526528 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.526385    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f9de60-s94gj\" (UID: \"0e8be54d-ca5b-4794-b516-7b14c36bc840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"
Nov 02 09:51:19.536587 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.536562    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhm2q\" (UniqueName: \"kubernetes.io/projected/0e8be54d-ca5b-4794-b516-7b14c36bc840-kube-api-access-xhm2q\") pod \"maas-api-revoke-sub-83f9de60-s94gj\" (UID: \"0e8be54d-ca5b-4794-b516-7b14c36bc840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"
Nov 02 09:51:19.680345 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:19.680307    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d92016_a072_4478_9231_12996763776e.slice/crio-a7cf5fbf9b0d4017ef56423fa516eec0a16dda1ee919949d33f599f3eff6bf00 WatchSource:0}: Error finding container a7cf5fbf9b0d4017ef56423fa516eec0a16dda1ee919949d33f599f3eff6bf00: Status 404 returned error can't find the container with id a7cf5fbf9b0d4017ef56423fa516eec0a16dda1ee919949d33f599f3eff6bf00
Nov 02 09:51:19.683533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.683514    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"]
Nov 02 09:51:19.707740 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:19.707707    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"
Nov 02 09:51:20.047740 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:20.047679    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8" event={"ID":"81d92016-a072-4478-9231-12996763776e","Type":"ContainerStarted","Data":"8befc71a565e8f871abc88c479de84d2a782190fa52169fb29a31b101d15df96"}
Nov 02 09:51:20.047928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:20.047823    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8" event={"ID":"81d92016-a072-4478-9231-12996763776e","Type":"ContainerStarted","Data":"a7cf5fbf9b0d4017ef56423fa516eec0a16dda1ee919949d33f599f3eff6bf00"}
Nov 02 09:51:20.048053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:20.048026    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:20.088529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:20.083406    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"]
Nov 02 09:51:20.088529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:20.085816    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7" event={"ID":"33be085e-851a-46a2-b44c-524abb75390a","Type":"ContainerStarted","Data":"174e7a9773bc330e9069451d0499ed2fe1ddef5fd042e847e141bf0f436bf6d6"}
Nov 02 09:51:20.088529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:20.085865    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7" event={"ID":"33be085e-851a-46a2-b44c-524abb75390a","Type":"ContainerStarted","Data":"63d4029a87ea297a48f42e811d2a9f9d75c7c6933a818adec1ac303001f26c89"}
Nov 02 09:51:20.088529 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:20.086759    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8be54d_ca5b_4794_b516_7b14c36bc840.slice/crio-fa7d09bffd5d14499bbceeeb40f3e8d52aeac656cae2df3addc404f8296d884d WatchSource:0}: Error finding container fa7d09bffd5d14499bbceeeb40f3e8d52aeac656cae2df3addc404f8296d884d: Status 404 returned error can't find the container with id fa7d09bffd5d14499bbceeeb40f3e8d52aeac656cae2df3addc404f8296d884d
Nov 02 09:51:20.094144 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:20.094090    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8" podStartSLOduration=2.094075085 podStartE2EDuration="2.094075085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:20.09069401 +0000 UTC m=+957.028980804" watchObservedRunningTime="2026-11-02 09:51:20.094075085 +0000 UTC m=+957.032361857"
Nov 02 09:51:20.125940 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:20.125878    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7" podStartSLOduration=2.125862585 podStartE2EDuration="2.125862585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:20.120536557 +0000 UTC m=+957.058823396" watchObservedRunningTime="2026-11-02 09:51:20.125862585 +0000 UTC m=+957.064149363"
Nov 02 09:51:21.110379 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:21.110340    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj" event={"ID":"0e8be54d-ca5b-4794-b516-7b14c36bc840","Type":"ContainerStarted","Data":"846cb1ec527facdd56f8f117d0f91519a87aa8ab7e7fdbe4794b18a709e7daff"}
Nov 02 09:51:21.110845 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:21.110410    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj" event={"ID":"0e8be54d-ca5b-4794-b516-7b14c36bc840","Type":"ContainerStarted","Data":"fa7d09bffd5d14499bbceeeb40f3e8d52aeac656cae2df3addc404f8296d884d"}
Nov 02 09:51:21.113791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:21.113743    4449 generic.go:356] "Generic (PLEG): container finished" podID="33be085e-851a-46a2-b44c-524abb75390a" containerID="174e7a9773bc330e9069451d0499ed2fe1ddef5fd042e847e141bf0f436bf6d6" exitCode=0
Nov 02 09:51:21.114789 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:21.114734    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7" event={"ID":"33be085e-851a-46a2-b44c-524abb75390a","Type":"ContainerDied","Data":"174e7a9773bc330e9069451d0499ed2fe1ddef5fd042e847e141bf0f436bf6d6"}
Nov 02 09:51:21.129664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:21.129615    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj" podStartSLOduration=2.129599621 podStartE2EDuration="2.129599621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:21.128823911 +0000 UTC m=+958.067110710" watchObservedRunningTime="2026-11-02 09:51:21.129599621 +0000 UTC m=+958.067886394"
Nov 02 09:51:22.124481 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.124442    4449 generic.go:356] "Generic (PLEG): container finished" podID="0e8be54d-ca5b-4794-b516-7b14c36bc840" containerID="846cb1ec527facdd56f8f117d0f91519a87aa8ab7e7fdbe4794b18a709e7daff" exitCode=0
Nov 02 09:51:22.124968 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.124532    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj" event={"ID":"0e8be54d-ca5b-4794-b516-7b14c36bc840","Type":"ContainerDied","Data":"846cb1ec527facdd56f8f117d0f91519a87aa8ab7e7fdbe4794b18a709e7daff"}
Nov 02 09:51:22.320091 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.320064    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"
Nov 02 09:51:22.395405 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.395331    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33be085e-851a-46a2-b44c-524abb75390a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33be085e-851a-46a2-b44c-524abb75390a-maas-api-openshift-ca\") pod \"33be085e-851a-46a2-b44c-524abb75390a\" (UID: \"33be085e-851a-46a2-b44c-524abb75390a\") "
Nov 02 09:51:22.395552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.395435    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33be085e-851a-46a2-b44c-524abb75390a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33be085e-851a-46a2-b44c-524abb75390a-maas-api-xks-ca\") pod \"33be085e-851a-46a2-b44c-524abb75390a\" (UID: \"33be085e-851a-46a2-b44c-524abb75390a\") "
Nov 02 09:51:22.395552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.395485    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33be085e-851a-46a2-b44c-524abb75390a-kube-api-access-cl652\" (UniqueName: \"kubernetes.io/projected/33be085e-851a-46a2-b44c-524abb75390a-kube-api-access-cl652\") pod \"33be085e-851a-46a2-b44c-524abb75390a\" (UID: \"33be085e-851a-46a2-b44c-524abb75390a\") "
Nov 02 09:51:22.395813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.395786    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33be085e-851a-46a2-b44c-524abb75390a-maas-api-openshift-ca" pod "33be085e-851a-46a2-b44c-524abb75390a" (UID: "33be085e-851a-46a2-b44c-524abb75390a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:22.395998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.395980    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33be085e-851a-46a2-b44c-524abb75390a-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:22.398492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.398457    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33be085e-851a-46a2-b44c-524abb75390a-maas-api-xks-ca" pod "33be085e-851a-46a2-b44c-524abb75390a" (UID: "33be085e-851a-46a2-b44c-524abb75390a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:51:22.398492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.398466    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33be085e-851a-46a2-b44c-524abb75390a-kube-api-access-cl652" pod "33be085e-851a-46a2-b44c-524abb75390a" (UID: "33be085e-851a-46a2-b44c-524abb75390a"). InnerVolumeSpecName "kube-api-access-cl652". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:22.497026 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.496990    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl652\" (UniqueName: \"kubernetes.io/projected/33be085e-851a-46a2-b44c-524abb75390a-kube-api-access-cl652\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:22.497026 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.497024    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33be085e-851a-46a2-b44c-524abb75390a-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:22.683524 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.683443    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"]
Nov 02 09:51:22.684169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.684154    4449 cpu_manager.go:414] "containerMap: removing container" podUID="33be085e-851a-46a2-b44c-524abb75390a" containerName="revoke-keys"
Nov 02 09:51:22.684255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.684170    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33be085e-851a-46a2-b44c-524abb75390a" containerName="revoke-keys"
Nov 02 09:51:22.684313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.684302    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="33be085e-851a-46a2-b44c-524abb75390a" containerName="revoke-keys"
Nov 02 09:51:22.688548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.688525    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.703392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.703367    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"]
Nov 02 09:51:22.801413 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.801373    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.801605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.801417    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkmc8\" (UniqueName: \"kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-kube-api-access-bkmc8\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.801605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.801511    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.801605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.801532    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.801605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.801581    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.801836 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.801615    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.801836 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.801718    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.801836 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.801743    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.801836 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.801779    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.904676 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.904639    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.904861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.904688    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.904861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.904751    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.904861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.904837    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.904969 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.904860    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkmc8\" (UniqueName: \"kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-kube-api-access-bkmc8\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.905000 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.904981    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.905031 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.905014    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.905090 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.905061    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.905197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.905172    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.905350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.905271    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.905430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.905356    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.905430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.905419    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.905552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.905444    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.906059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.906035    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.907705 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.907676    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.908254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.908203    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.918618 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.918587    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:22.918797 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:22.918771    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkmc8\" (UniqueName: \"kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-kube-api-access-bkmc8\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:23.008170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.008086    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:23.136941 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.136910    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"
Nov 02 09:51:23.137446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.136960    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7" event={"ID":"33be085e-851a-46a2-b44c-524abb75390a","Type":"ContainerDied","Data":"63d4029a87ea297a48f42e811d2a9f9d75c7c6933a818adec1ac303001f26c89"}
Nov 02 09:51:23.137446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.137006    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63d4029a87ea297a48f42e811d2a9f9d75c7c6933a818adec1ac303001f26c89"
Nov 02 09:51:23.312076 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:23.312030    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a81758_9db8_4221_9ab7_7a20d2400a2d.slice/crio-36aaa463c3cd50868d0bcb8feffe4a0da61d6126ffd20544f95eb3c41c7b479d WatchSource:0}: Error finding container 36aaa463c3cd50868d0bcb8feffe4a0da61d6126ffd20544f95eb3c41c7b479d: Status 404 returned error can't find the container with id 36aaa463c3cd50868d0bcb8feffe4a0da61d6126ffd20544f95eb3c41c7b479d
Nov 02 09:51:23.314899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.314861    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:51:23.315007 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.314935    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:51:23.315007 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.314962    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:51:23.315007 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.314876    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"]
Nov 02 09:51:23.434963 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.434716    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"
Nov 02 09:51:23.515192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.513843    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8be54d-ca5b-4794-b516-7b14c36bc840-kube-api-access-xhm2q\" (UniqueName: \"kubernetes.io/projected/0e8be54d-ca5b-4794-b516-7b14c36bc840-kube-api-access-xhm2q\") pod \"0e8be54d-ca5b-4794-b516-7b14c36bc840\" (UID: \"0e8be54d-ca5b-4794-b516-7b14c36bc840\") "
Nov 02 09:51:23.515192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.513992    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-openshift-ca\") pod \"0e8be54d-ca5b-4794-b516-7b14c36bc840\" (UID: \"0e8be54d-ca5b-4794-b516-7b14c36bc840\") "
Nov 02 09:51:23.515192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.514168    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-xks-ca\") pod \"0e8be54d-ca5b-4794-b516-7b14c36bc840\" (UID: \"0e8be54d-ca5b-4794-b516-7b14c36bc840\") "
Nov 02 09:51:23.523260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.519557    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-openshift-ca" pod "0e8be54d-ca5b-4794-b516-7b14c36bc840" (UID: "0e8be54d-ca5b-4794-b516-7b14c36bc840"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:23.525438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.525382    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8be54d-ca5b-4794-b516-7b14c36bc840-kube-api-access-xhm2q" pod "0e8be54d-ca5b-4794-b516-7b14c36bc840" (UID: "0e8be54d-ca5b-4794-b516-7b14c36bc840"). InnerVolumeSpecName "kube-api-access-xhm2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:23.544935 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.544498    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-xks-ca" pod "0e8be54d-ca5b-4794-b516-7b14c36bc840" (UID: "0e8be54d-ca5b-4794-b516-7b14c36bc840"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:51:23.556112 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.556025    4449 scope.go:122] "RemoveContainer" containerID="cb551ad31dae561e6989f82334b51c8ec0fc861ba2809c15d14aa2133ebc86b5"
Nov 02 09:51:23.574414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.573710    4449 scope.go:122] "RemoveContainer" containerID="ebc820c17c8afbe794d691337d245e870fcef4ff9c0ba9a18e6469b33aca93db"
Nov 02 09:51:23.616018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.615983    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhm2q\" (UniqueName: \"kubernetes.io/projected/0e8be54d-ca5b-4794-b516-7b14c36bc840-kube-api-access-xhm2q\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:23.616018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.616019    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:23.616271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:23.616034    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8be54d-ca5b-4794-b516-7b14c36bc840-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:24.009437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:24.009353    4449 patch_prober.go:29] interesting pod/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:51:24.009585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:24.009440    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:24.150980 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:24.150944    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj" event={"ID":"0e8be54d-ca5b-4794-b516-7b14c36bc840","Type":"ContainerDied","Data":"fa7d09bffd5d14499bbceeeb40f3e8d52aeac656cae2df3addc404f8296d884d"}
Nov 02 09:51:24.151603 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:24.151582    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa7d09bffd5d14499bbceeeb40f3e8d52aeac656cae2df3addc404f8296d884d"
Nov 02 09:51:24.151790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:24.151778    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"
Nov 02 09:51:24.159546 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:24.159507    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" event={"ID":"a0a81758-9db8-4221-9ab7-7a20d2400a2d","Type":"ContainerStarted","Data":"45373cd1ec8060d63bf6c6f42240f34ed8de3daab98beb28700607f02cb0ff4f"}
Nov 02 09:51:24.159792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:24.159770    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" event={"ID":"a0a81758-9db8-4221-9ab7-7a20d2400a2d","Type":"ContainerStarted","Data":"36aaa463c3cd50868d0bcb8feffe4a0da61d6126ffd20544f95eb3c41c7b479d"}
Nov 02 09:51:25.008483 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:25.008450    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:25.009854 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:25.009829    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 09:51:25.009951 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:25.009880    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 09:51:26.008783 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.008752    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 09:51:26.009281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.008806    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 09:51:26.192767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.192686    4449 generic.go:356] "Generic (PLEG): container finished" podID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerID="2f27071bfe5e8042abf5139c2229932e2326959715a89dec6a852cc70040987a" exitCode=0
Nov 02 09:51:26.192914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.192764    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" event={"ID":"22f0ca6c-1150-4c10-9510-3613a96e425c","Type":"ContainerDied","Data":"2f27071bfe5e8042abf5139c2229932e2326959715a89dec6a852cc70040987a"}
Nov 02 09:51:26.193332 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.193313    4449 scope.go:122] "RemoveContainer" containerID="2f27071bfe5e8042abf5139c2229932e2326959715a89dec6a852cc70040987a"
Nov 02 09:51:26.194779 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.194739    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:51:26.194908 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.194820    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:51:26.194908 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.194862    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:51:26.219384 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.219344    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" podStartSLOduration=4.21932977 podStartE2EDuration="4.21932977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:24.195584187 +0000 UTC m=+961.133870960" watchObservedRunningTime="2026-11-02 09:51:26.21932977 +0000 UTC m=+963.157616578"
Nov 02 09:51:26.316995 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.316906    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"]
Nov 02 09:51:26.318150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.318127    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0e8be54d-ca5b-4794-b516-7b14c36bc840" containerName="revoke-keys"
Nov 02 09:51:26.318237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.318154    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8be54d-ca5b-4794-b516-7b14c36bc840" containerName="revoke-keys"
Nov 02 09:51:26.318379 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.318366    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8be54d-ca5b-4794-b516-7b14c36bc840" containerName="revoke-keys"
Nov 02 09:51:26.322267 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.321622    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"
Nov 02 09:51:26.324819 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.324794    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"]
Nov 02 09:51:26.328581 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.327882    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:51:26.328581 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.327981    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:51:26.330207 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.330183    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:51:26.330419 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.330347    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:51:26.399784 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.399749    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4893be-42r6d\" (UID: \"7a7fefd8-6d22-464a-890d-40983e8e9288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"
Nov 02 09:51:26.399972 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.399813    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4893be-42r6d\" (UID: \"7a7fefd8-6d22-464a-890d-40983e8e9288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"
Nov 02 09:51:26.400033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.399998    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf262\" (UniqueName: \"kubernetes.io/projected/7a7fefd8-6d22-464a-890d-40983e8e9288-kube-api-access-kf262\") pod \"maas-api-revoke-sub-eb4893be-42r6d\" (UID: \"7a7fefd8-6d22-464a-890d-40983e8e9288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"
Nov 02 09:51:26.501087 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.501008    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf262\" (UniqueName: \"kubernetes.io/projected/7a7fefd8-6d22-464a-890d-40983e8e9288-kube-api-access-kf262\") pod \"maas-api-revoke-sub-eb4893be-42r6d\" (UID: \"7a7fefd8-6d22-464a-890d-40983e8e9288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"
Nov 02 09:51:26.501270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.501190    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4893be-42r6d\" (UID: \"7a7fefd8-6d22-464a-890d-40983e8e9288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"
Nov 02 09:51:26.501270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.501256    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4893be-42r6d\" (UID: \"7a7fefd8-6d22-464a-890d-40983e8e9288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"
Nov 02 09:51:26.501867 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.501843    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4893be-42r6d\" (UID: \"7a7fefd8-6d22-464a-890d-40983e8e9288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"
Nov 02 09:51:26.503897 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.503870    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4893be-42r6d\" (UID: \"7a7fefd8-6d22-464a-890d-40983e8e9288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"
Nov 02 09:51:26.513295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.513271    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf262\" (UniqueName: \"kubernetes.io/projected/7a7fefd8-6d22-464a-890d-40983e8e9288-kube-api-access-kf262\") pod \"maas-api-revoke-sub-eb4893be-42r6d\" (UID: \"7a7fefd8-6d22-464a-890d-40983e8e9288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"
Nov 02 09:51:26.692592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:26.692543    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"
Nov 02 09:51:27.009241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:27.008656    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 09:51:27.009241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:27.008724    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 09:51:27.311804 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:27.311763    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" event={"ID":"22f0ca6c-1150-4c10-9510-3613a96e425c","Type":"ContainerStarted","Data":"609c052c784857f1319247abe88a9263dd3abba05768b2ade2439c78f0e0edfd"}
Nov 02 09:51:27.934695 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:27.934662    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:51:27.935032 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:27.935010    4449 patch_prober.go:29] interesting pod/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 09:51:27.935179 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:27.935155    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 09:51:28.013247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:28.012751    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 09:51:28.013247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:28.012814    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 09:51:28.079242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:28.063652    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 09:51:28.083258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:28.080547    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:28.083258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:28.082567    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": context deadline exceeded"
Nov 02 09:51:28.144488 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:28.142377    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7fefd8_6d22_464a_890d_40983e8e9288.slice/crio-3156cc3c239670c705961d2cd07f8297edf5b0d9cdeb8de459bd6ef652656fc5 WatchSource:0}: Error finding container 3156cc3c239670c705961d2cd07f8297edf5b0d9cdeb8de459bd6ef652656fc5: Status 404 returned error can't find the container with id 3156cc3c239670c705961d2cd07f8297edf5b0d9cdeb8de459bd6ef652656fc5
Nov 02 09:51:28.156494 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:28.156453    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"]
Nov 02 09:51:28.574406 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:28.574327    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:28.586279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:28.586244    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d" event={"ID":"7a7fefd8-6d22-464a-890d-40983e8e9288","Type":"ContainerStarted","Data":"3156cc3c239670c705961d2cd07f8297edf5b0d9cdeb8de459bd6ef652656fc5"}
Nov 02 09:51:28.930632 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:28.930592    4449 patch_prober.go:29] interesting pod/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 09:51:28.931687 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:28.931279    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 09:51:29.009035 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:29.008592    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 09:51:29.009035 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:29.008653    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 09:51:29.066543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:29.066446    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:29.072243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:29.067753    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:51:29.083244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:29.082133    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:29.083244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:29.082513    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:51:29.690265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:29.677892    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": context deadline exceeded"
Nov 02 09:51:29.690265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:29.678661    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:51:29.715535 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:29.697291    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"]
Nov 02 09:51:29.756244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:29.751640    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:29.965319 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:29.951376    4449 patch_prober.go:29] interesting pod/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 09:51:29.965319 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:29.951429    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 09:51:29.966632 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:29.966597    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:30.024345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.024204    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 09:51:30.024345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.024285    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 09:51:30.249943 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.249841    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 09:51:30.254242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.250697    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"]
Nov 02 09:51:30.268151 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.268103    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:30.325244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.298030    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 02 09:51:30.754888 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.746392    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:30.963246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.798377    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded"
Nov 02 09:51:30.963246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.887536    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:51:30.963246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.887583    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d" event={"ID":"7a7fefd8-6d22-464a-890d-40983e8e9288","Type":"ContainerStarted","Data":"59ac81c3a7a970f88fb84d91bb749a225c03f13c11f9780cecd1c3bfb9e7cd94"}
Nov 02 09:51:30.963246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.890160    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:30.963246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.898752    4449 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/health\": context deadline exceeded"
Nov 02 09:51:30.967243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.966042    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:30.967243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.966102    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:30.967243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.966324    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9dlr\" (UniqueName: \"kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-kube-api-access-r9dlr\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:30.967243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:30.966396    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:31.098261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.014948    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 09:51:31.098261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.015022    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 09:51:31.098261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.027897    4449 patch_prober.go:29] interesting pod/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 09:51:31.098261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.027948    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 09:51:31.220103 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.220066    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"]
Nov 02 09:51:31.234262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.222890    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:31.234262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.222944    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:31.234262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.223112    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9dlr\" (UniqueName: \"kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-kube-api-access-r9dlr\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:31.234262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.223183    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:31.285271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.239948    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:31.285271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.242781    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:31.285271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.271662    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 09:51:31.310585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.310555    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-3337a4c7a0e4\""
Nov 02 09:51:31.487403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.487355    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:31.495780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.495742    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": context deadline exceeded"
Nov 02 09:51:31.891380 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.891340    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": context deadline exceeded"
Nov 02 09:51:31.973275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.965675    4449 patch_prober.go:29] interesting pod/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 09:51:31.973275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.965745    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 09:51:31.973275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:31.966501    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:32.167030 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.166932    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:32.171719 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.171675    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9dlr\" (UniqueName: \"kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-kube-api-access-r9dlr\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:32.172766 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.172723    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 09:51:32.173357 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.173329    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 09:51:32.189725 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.189698    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:51:32.190006 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.189989    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 09:51:32.191063 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.191045    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:51:32.193783 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.193761    4449 generic.go:356] "Generic (PLEG): container finished" podID="7a7fefd8-6d22-464a-890d-40983e8e9288" containerID="59ac81c3a7a970f88fb84d91bb749a225c03f13c11f9780cecd1c3bfb9e7cd94" exitCode=0
Nov 02 09:51:32.195018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.194992    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d" event={"ID":"7a7fefd8-6d22-464a-890d-40983e8e9288","Type":"ContainerDied","Data":"59ac81c3a7a970f88fb84d91bb749a225c03f13c11f9780cecd1c3bfb9e7cd94"}
Nov 02 09:51:32.296383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.296248    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:32.452879 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.452797    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:32.496396 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.496330    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:32.533741 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.532870    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"]
Nov 02 09:51:32.534322 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.534300    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:51:32.567275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.567243    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:51:32.626254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.625462    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:51:32.632657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.632628    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"]
Nov 02 09:51:32.664107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.664039    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"]
Nov 02 09:51:32.713337 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.710259    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8bfdd0-vrr4j"]
Nov 02 09:51:32.719679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.719558    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d" podStartSLOduration=6.719539088 podStartE2EDuration="6.719539088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:30.314014276 +0000 UTC m=+967.252301053" watchObservedRunningTime="2026-11-02 09:51:32.719539088 +0000 UTC m=+969.657825866"
Nov 02 09:51:32.743151 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.742884    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"]
Nov 02 09:51:32.764303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.760430    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"
Nov 02 09:51:32.781286 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.771755    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt6r9\" (UniqueName: \"kubernetes.io/projected/c4674912-6f57-42d8-b948-a36fd412d1b7-kube-api-access-jt6r9\") pod \"maas-api-revoke-sub-0f6d6713-9cnnl\" (UID: \"c4674912-6f57-42d8-b948-a36fd412d1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"
Nov 02 09:51:32.828121 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.820332    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f6d6713-9cnnl\" (UID: \"c4674912-6f57-42d8-b948-a36fd412d1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"
Nov 02 09:51:32.828121 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.820396    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f6d6713-9cnnl\" (UID: \"c4674912-6f57-42d8-b948-a36fd412d1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"
Nov 02 09:51:32.886243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.874930    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"]
Nov 02 09:51:32.921996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.921743    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f6d6713-9cnnl\" (UID: \"c4674912-6f57-42d8-b948-a36fd412d1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"
Nov 02 09:51:32.921996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.921802    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f6d6713-9cnnl\" (UID: \"c4674912-6f57-42d8-b948-a36fd412d1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"
Nov 02 09:51:32.921996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.921984    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt6r9\" (UniqueName: \"kubernetes.io/projected/c4674912-6f57-42d8-b948-a36fd412d1b7-kube-api-access-jt6r9\") pod \"maas-api-revoke-sub-0f6d6713-9cnnl\" (UID: \"c4674912-6f57-42d8-b948-a36fd412d1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"
Nov 02 09:51:32.923115 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.923058    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f6d6713-9cnnl\" (UID: \"c4674912-6f57-42d8-b948-a36fd412d1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"
Nov 02 09:51:32.925386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.925362    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f6d6713-9cnnl\" (UID: \"c4674912-6f57-42d8-b948-a36fd412d1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"
Nov 02 09:51:32.929471 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.929449    4449 patch_prober.go:29] interesting pod/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 09:51:32.929571 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.929514    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 09:51:32.935755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:32.935712    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt6r9\" (UniqueName: \"kubernetes.io/projected/c4674912-6f57-42d8-b948-a36fd412d1b7-kube-api-access-jt6r9\") pod \"maas-api-revoke-sub-0f6d6713-9cnnl\" (UID: \"c4674912-6f57-42d8-b948-a36fd412d1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"
Nov 02 09:51:33.008750 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.008634    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 09:51:33.008750 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.008706    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 09:51:33.074251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.074183    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"]
Nov 02 09:51:33.090766 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:33.090730    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7872390_6d52_4e22_a7dd_0617bf443ec0.slice/crio-3770bc4819e7d6109af35f6f6622ee360cf235ce293f0f174534f97ef6ff51e9 WatchSource:0}: Error finding container 3770bc4819e7d6109af35f6f6622ee360cf235ce293f0f174534f97ef6ff51e9: Status 404 returned error can't find the container with id 3770bc4819e7d6109af35f6f6622ee360cf235ce293f0f174534f97ef6ff51e9
Nov 02 09:51:33.212962 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.212923    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"
Nov 02 09:51:33.221519 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.221488    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af5227c4-b584-4416-86c2-8a94d61db2cd" path="/var/lib/kubelet/pods/af5227c4-b584-4416-86c2-8a94d61db2cd/volumes"
Nov 02 09:51:33.222303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.222276    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:33.222394 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.222356    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb" event={"ID":"b7872390-6d52-4e22-a7dd-0617bf443ec0","Type":"ContainerStarted","Data":"3e96954ac4229fde7fade54ad17ac11b1e1cf95081145392e210979b533b2841"}
Nov 02 09:51:33.222452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.222391    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb" event={"ID":"b7872390-6d52-4e22-a7dd-0617bf443ec0","Type":"ContainerStarted","Data":"3770bc4819e7d6109af35f6f6622ee360cf235ce293f0f174534f97ef6ff51e9"}
Nov 02 09:51:33.247368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.247292    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb" podStartSLOduration=4.247271918 podStartE2EDuration="4.247271918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:33.24519983 +0000 UTC m=+970.183486651" watchObservedRunningTime="2026-11-02 09:51:33.247271918 +0000 UTC m=+970.185558694"
Nov 02 09:51:33.514826 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.514798    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"
Nov 02 09:51:33.590575 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:33.590543    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4674912_6f57_42d8_b948_a36fd412d1b7.slice/crio-021daa025f183c0e00a2c07bcd41de209e60ccd210ecaa2cef7905c3aa8f9a7a WatchSource:0}: Error finding container 021daa025f183c0e00a2c07bcd41de209e60ccd210ecaa2cef7905c3aa8f9a7a: Status 404 returned error can't find the container with id 021daa025f183c0e00a2c07bcd41de209e60ccd210ecaa2cef7905c3aa8f9a7a
Nov 02 09:51:33.594749 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.594711    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"]
Nov 02 09:51:33.639736 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.639709    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7fefd8-6d22-464a-890d-40983e8e9288-kube-api-access-kf262\" (UniqueName: \"kubernetes.io/projected/7a7fefd8-6d22-464a-890d-40983e8e9288-kube-api-access-kf262\") pod \"7a7fefd8-6d22-464a-890d-40983e8e9288\" (UID: \"7a7fefd8-6d22-464a-890d-40983e8e9288\") "
Nov 02 09:51:33.639894 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.639752    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-openshift-ca\") pod \"7a7fefd8-6d22-464a-890d-40983e8e9288\" (UID: \"7a7fefd8-6d22-464a-890d-40983e8e9288\") "
Nov 02 09:51:33.639894 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.639806    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-xks-ca\") pod \"7a7fefd8-6d22-464a-890d-40983e8e9288\" (UID: \"7a7fefd8-6d22-464a-890d-40983e8e9288\") "
Nov 02 09:51:33.640284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.640253    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-openshift-ca" pod "7a7fefd8-6d22-464a-890d-40983e8e9288" (UID: "7a7fefd8-6d22-464a-890d-40983e8e9288"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:33.640770 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.640748    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:33.642096 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.642067    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7fefd8-6d22-464a-890d-40983e8e9288-kube-api-access-kf262" pod "7a7fefd8-6d22-464a-890d-40983e8e9288" (UID: "7a7fefd8-6d22-464a-890d-40983e8e9288"). InnerVolumeSpecName "kube-api-access-kf262". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:33.642485 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.642464    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-xks-ca" pod "7a7fefd8-6d22-464a-890d-40983e8e9288" (UID: "7a7fefd8-6d22-464a-890d-40983e8e9288"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:51:33.742124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.742088    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf262\" (UniqueName: \"kubernetes.io/projected/7a7fefd8-6d22-464a-890d-40983e8e9288-kube-api-access-kf262\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:33.742124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.742120    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7fefd8-6d22-464a-890d-40983e8e9288-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:33.929269 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.928781    4449 patch_prober.go:29] interesting pod/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 09:51:33.929269 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.928848    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 09:51:33.944482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.944451    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"]
Nov 02 09:51:33.945353 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.945335    4449 cpu_manager.go:414] "containerMap: removing container" podUID="7a7fefd8-6d22-464a-890d-40983e8e9288" containerName="revoke-keys"
Nov 02 09:51:33.945353 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.945353    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7fefd8-6d22-464a-890d-40983e8e9288" containerName="revoke-keys"
Nov 02 09:51:33.945488 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.945477    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7fefd8-6d22-464a-890d-40983e8e9288" containerName="revoke-keys"
Nov 02 09:51:33.984294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.981854    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:33.986088 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.986048    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9cbd331c-openshift-default-dockercfg-t4z5f\""
Nov 02 09:51:33.988743 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:33.986901    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"]
Nov 02 09:51:34.015395 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.015198    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:34.016348 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.016288    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:34.017160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.017096    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:51:34.046824 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.046785    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-data\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.047002 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.046921    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-token\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.047002 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.046964    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istiod-ca-cert\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.047091 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.047024    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-credential-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.047091 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.047054    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.047194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.047093    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-certs\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.047354 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.047329    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqzhk\" (UniqueName: \"kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-kube-api-access-nqzhk\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.047437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.047370    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-podinfo\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.047498 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.047431    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-envoy\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.148346 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.148259    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-data\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.148530 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.148359    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-token\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.148530 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.148415    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istiod-ca-cert\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.148530 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.148512    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-credential-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.148724 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.148553    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.148724 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.148580    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-certs\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.148839 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.148738    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqzhk\" (UniqueName: \"kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-kube-api-access-nqzhk\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.148839 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.148741    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-data\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.148839 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.148767    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-podinfo\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.148839 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.148799    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-envoy\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.149061 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.148988    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-credential-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.157391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.149633    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istiod-ca-cert\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.178276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.176720    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.178276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.176876    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-certs\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.180119 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.180088    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-podinfo\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.187750 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.187719    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-envoy\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.187885 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.187849    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-token\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.189460 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.189433    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqzhk\" (UniqueName: \"kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-kube-api-access-nqzhk\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.226453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.226425    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"
Nov 02 09:51:34.226644 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.226466    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d" event={"ID":"7a7fefd8-6d22-464a-890d-40983e8e9288","Type":"ContainerDied","Data":"3156cc3c239670c705961d2cd07f8297edf5b0d9cdeb8de459bd6ef652656fc5"}
Nov 02 09:51:34.226644 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.226518    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3156cc3c239670c705961d2cd07f8297edf5b0d9cdeb8de459bd6ef652656fc5"
Nov 02 09:51:34.228226 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.228185    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl" event={"ID":"c4674912-6f57-42d8-b948-a36fd412d1b7","Type":"ContainerStarted","Data":"0c55d35abf7ea2f880c7a62ef88c56893baf885506f64fb2242be446d5c8ca1a"}
Nov 02 09:51:34.228348 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.228250    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl" event={"ID":"c4674912-6f57-42d8-b948-a36fd412d1b7","Type":"ContainerStarted","Data":"021daa025f183c0e00a2c07bcd41de209e60ccd210ecaa2cef7905c3aa8f9a7a"}
Nov 02 09:51:34.286245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.282885    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl" podStartSLOduration=3.2828640350000002 podStartE2EDuration="3.282864035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:34.247501912 +0000 UTC m=+971.185788688" watchObservedRunningTime="2026-11-02 09:51:34.282864035 +0000 UTC m=+971.221150811"
Nov 02 09:51:34.317242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.311492    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:34.698421 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.698306    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8" podUID="81d92016-a072-4478-9231-12996763776e" containerName="payload-processing" containerID="cri-o://8befc71a565e8f871abc88c479de84d2a782190fa52169fb29a31b101d15df96" gracePeriod=25
Nov 02 09:51:34.744982 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.744951    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"]
Nov 02 09:51:34.818338 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:34.818290    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f4c5ab_d3da_449e_b5dc_d6fdcf0751cc.slice/crio-be71c842a63ef67cde1b312242430ac9908601f607ccc3887917848a355b3f3b WatchSource:0}: Error finding container be71c842a63ef67cde1b312242430ac9908601f607ccc3887917848a355b3f3b: Status 404 returned error can't find the container with id be71c842a63ef67cde1b312242430ac9908601f607ccc3887917848a355b3f3b
Nov 02 09:51:34.821128 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.821091    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:51:34.821277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.821183    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:51:34.821277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.821265    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:51:34.934273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.929468    4449 patch_prober.go:29] interesting pod/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 09:51:34.934273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:34.929571    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 09:51:35.027824 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.027798    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:35.124364 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.124333    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-plugins-config-volume\") pod \"81d92016-a072-4478-9231-12996763776e\" (UID: \"81d92016-a072-4478-9231-12996763776e\") "
Nov 02 09:51:35.124557 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.124404    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-provider-credentials\") pod \"81d92016-a072-4478-9231-12996763776e\" (UID: \"81d92016-a072-4478-9231-12996763776e\") "
Nov 02 09:51:35.124637 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.124575    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-kube-api-access-4d27p\" (UniqueName: \"kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-kube-api-access-4d27p\") pod \"81d92016-a072-4478-9231-12996763776e\" (UID: \"81d92016-a072-4478-9231-12996763776e\") "
Nov 02 09:51:35.124688 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.124661    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-routing-overlay-volume\") pod \"81d92016-a072-4478-9231-12996763776e\" (UID: \"81d92016-a072-4478-9231-12996763776e\") "
Nov 02 09:51:35.127270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.124813    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-plugins-config-volume" pod "81d92016-a072-4478-9231-12996763776e" (UID: "81d92016-a072-4478-9231-12996763776e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:35.127270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.125514    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-routing-overlay-volume" pod "81d92016-a072-4478-9231-12996763776e" (UID: "81d92016-a072-4478-9231-12996763776e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:35.127270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.126258    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:35.127270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.126288    4449 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81d92016-a072-4478-9231-12996763776e-routing-overlay-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:35.137438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.137407    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-kube-api-access-4d27p" pod "81d92016-a072-4478-9231-12996763776e" (UID: "81d92016-a072-4478-9231-12996763776e"). InnerVolumeSpecName "kube-api-access-4d27p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:35.142569 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.142540    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-provider-credentials" pod "81d92016-a072-4478-9231-12996763776e" (UID: "81d92016-a072-4478-9231-12996763776e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:35.228794 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.228764    4449 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-provider-credentials\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:35.228922 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.228797    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d27p\" (UniqueName: \"kubernetes.io/projected/81d92016-a072-4478-9231-12996763776e-kube-api-access-4d27p\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:35.240228 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.240178    4449 generic.go:356] "Generic (PLEG): container finished" podID="81d92016-a072-4478-9231-12996763776e" containerID="8befc71a565e8f871abc88c479de84d2a782190fa52169fb29a31b101d15df96" exitCode=0
Nov 02 09:51:35.240453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.240267    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"
Nov 02 09:51:35.240453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.240284    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8" event={"ID":"81d92016-a072-4478-9231-12996763776e","Type":"ContainerDied","Data":"8befc71a565e8f871abc88c479de84d2a782190fa52169fb29a31b101d15df96"}
Nov 02 09:51:35.240453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.240352    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8" event={"ID":"81d92016-a072-4478-9231-12996763776e","Type":"ContainerDied","Data":"a7cf5fbf9b0d4017ef56423fa516eec0a16dda1ee919949d33f599f3eff6bf00"}
Nov 02 09:51:35.240453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.240382    4449 scope.go:122] "RemoveContainer" containerID="8befc71a565e8f871abc88c479de84d2a782190fa52169fb29a31b101d15df96"
Nov 02 09:51:35.242736 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.242571    4449 generic.go:356] "Generic (PLEG): container finished" podID="c4674912-6f57-42d8-b948-a36fd412d1b7" containerID="0c55d35abf7ea2f880c7a62ef88c56893baf885506f64fb2242be446d5c8ca1a" exitCode=0
Nov 02 09:51:35.242736 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.242680    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl" event={"ID":"c4674912-6f57-42d8-b948-a36fd412d1b7","Type":"ContainerDied","Data":"0c55d35abf7ea2f880c7a62ef88c56893baf885506f64fb2242be446d5c8ca1a"}
Nov 02 09:51:35.244818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.244749    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" event={"ID":"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc","Type":"ContainerStarted","Data":"85827e0be84c335ba8f2d9ee3460f4aa50669a03f14e148a739074408c35331b"}
Nov 02 09:51:35.244818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.244797    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" event={"ID":"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc","Type":"ContainerStarted","Data":"be71c842a63ef67cde1b312242430ac9908601f607ccc3887917848a355b3f3b"}
Nov 02 09:51:35.270466 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.270440    4449 scope.go:122] "RemoveContainer" containerID="8befc71a565e8f871abc88c479de84d2a782190fa52169fb29a31b101d15df96"
Nov 02 09:51:35.273255 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:51:35.270842    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8befc71a565e8f871abc88c479de84d2a782190fa52169fb29a31b101d15df96\": container with ID starting with 8befc71a565e8f871abc88c479de84d2a782190fa52169fb29a31b101d15df96 not found: ID does not exist" containerID="8befc71a565e8f871abc88c479de84d2a782190fa52169fb29a31b101d15df96"
Nov 02 09:51:35.273255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.270883    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8befc71a565e8f871abc88c479de84d2a782190fa52169fb29a31b101d15df96"} err="failed to get container status \"8befc71a565e8f871abc88c479de84d2a782190fa52169fb29a31b101d15df96\": rpc error: code = NotFound desc = could not find container \"8befc71a565e8f871abc88c479de84d2a782190fa52169fb29a31b101d15df96\": container with ID starting with 8befc71a565e8f871abc88c479de84d2a782190fa52169fb29a31b101d15df96 not found: ID does not exist"
Nov 02 09:51:35.310781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.310717    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" podStartSLOduration=2.31069866 podStartE2EDuration="2.31069866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:35.310021772 +0000 UTC m=+972.248308549" watchObservedRunningTime="2026-11-02 09:51:35.31069866 +0000 UTC m=+972.248985437"
Nov 02 09:51:35.314769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.314731    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:35.316253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.316194    4449 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 02 09:51:35.316352 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.316296    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 02 09:51:35.379930 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.379890    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"]
Nov 02 09:51:35.385386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.385357    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f8995glm8"]
Nov 02 09:51:35.929555 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.929520    4449 patch_prober.go:29] interesting pod/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 09:51:35.929979 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:35.929587    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 09:51:36.320501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.320276    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:36.455596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.455564    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"
Nov 02 09:51:36.527285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.527191    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"]
Nov 02 09:51:36.527967 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.527948    4449 cpu_manager.go:414] "containerMap: removing container" podUID="c4674912-6f57-42d8-b948-a36fd412d1b7" containerName="revoke-keys"
Nov 02 09:51:36.528046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.527969    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4674912-6f57-42d8-b948-a36fd412d1b7" containerName="revoke-keys"
Nov 02 09:51:36.528046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.528015    4449 cpu_manager.go:414] "containerMap: removing container" podUID="81d92016-a072-4478-9231-12996763776e" containerName="payload-processing"
Nov 02 09:51:36.528147 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.528043    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d92016-a072-4478-9231-12996763776e" containerName="payload-processing"
Nov 02 09:51:36.528414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.528389    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d92016-a072-4478-9231-12996763776e" containerName="payload-processing"
Nov 02 09:51:36.528539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.528436    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4674912-6f57-42d8-b948-a36fd412d1b7" containerName="revoke-keys"
Nov 02 09:51:36.538315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.536137    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"
Nov 02 09:51:36.547547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.546768    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"]
Nov 02 09:51:36.554241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.554040    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4674912-6f57-42d8-b948-a36fd412d1b7-kube-api-access-jt6r9\" (UniqueName: \"kubernetes.io/projected/c4674912-6f57-42d8-b948-a36fd412d1b7-kube-api-access-jt6r9\") pod \"c4674912-6f57-42d8-b948-a36fd412d1b7\" (UID: \"c4674912-6f57-42d8-b948-a36fd412d1b7\") "
Nov 02 09:51:36.554241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.554116    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-openshift-ca\") pod \"c4674912-6f57-42d8-b948-a36fd412d1b7\" (UID: \"c4674912-6f57-42d8-b948-a36fd412d1b7\") "
Nov 02 09:51:36.554406 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.554326    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-xks-ca\") pod \"c4674912-6f57-42d8-b948-a36fd412d1b7\" (UID: \"c4674912-6f57-42d8-b948-a36fd412d1b7\") "
Nov 02 09:51:36.563314 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.559129    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4674912-6f57-42d8-b948-a36fd412d1b7-kube-api-access-jt6r9" pod "c4674912-6f57-42d8-b948-a36fd412d1b7" (UID: "c4674912-6f57-42d8-b948-a36fd412d1b7"). InnerVolumeSpecName "kube-api-access-jt6r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:36.563314 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.559411    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-openshift-ca" pod "c4674912-6f57-42d8-b948-a36fd412d1b7" (UID: "c4674912-6f57-42d8-b948-a36fd412d1b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:36.570700 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.570431    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-xks-ca" pod "c4674912-6f57-42d8-b948-a36fd412d1b7" (UID: "c4674912-6f57-42d8-b948-a36fd412d1b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:51:36.570700 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:51:36.570669    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:51:36.656682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.656643    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b9baa7e-qvmrc\" (UID: \"6a524abe-238d-4fcf-a096-d07032f33da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"
Nov 02 09:51:36.656848 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.656704    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgkv\" (UniqueName: \"kubernetes.io/projected/6a524abe-238d-4fcf-a096-d07032f33da4-kube-api-access-gmgkv\") pod \"maas-api-revoke-sub-6b9baa7e-qvmrc\" (UID: \"6a524abe-238d-4fcf-a096-d07032f33da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"
Nov 02 09:51:36.656981 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.656956    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b9baa7e-qvmrc\" (UID: \"6a524abe-238d-4fcf-a096-d07032f33da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"
Nov 02 09:51:36.657083 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.657068    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt6r9\" (UniqueName: \"kubernetes.io/projected/c4674912-6f57-42d8-b948-a36fd412d1b7-kube-api-access-jt6r9\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:36.657134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.657088    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:36.657134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.657116    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4674912-6f57-42d8-b948-a36fd412d1b7-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:36.758716 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.758680    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b9baa7e-qvmrc\" (UID: \"6a524abe-238d-4fcf-a096-d07032f33da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"
Nov 02 09:51:36.758901 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.758731    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b9baa7e-qvmrc\" (UID: \"6a524abe-238d-4fcf-a096-d07032f33da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"
Nov 02 09:51:36.758901 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.758768    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgkv\" (UniqueName: \"kubernetes.io/projected/6a524abe-238d-4fcf-a096-d07032f33da4-kube-api-access-gmgkv\") pod \"maas-api-revoke-sub-6b9baa7e-qvmrc\" (UID: \"6a524abe-238d-4fcf-a096-d07032f33da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"
Nov 02 09:51:36.759464 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.759437    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b9baa7e-qvmrc\" (UID: \"6a524abe-238d-4fcf-a096-d07032f33da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"
Nov 02 09:51:36.761159 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.761138    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b9baa7e-qvmrc\" (UID: \"6a524abe-238d-4fcf-a096-d07032f33da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"
Nov 02 09:51:36.771475 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.771450    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgkv\" (UniqueName: \"kubernetes.io/projected/6a524abe-238d-4fcf-a096-d07032f33da4-kube-api-access-gmgkv\") pod \"maas-api-revoke-sub-6b9baa7e-qvmrc\" (UID: \"6a524abe-238d-4fcf-a096-d07032f33da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"
Nov 02 09:51:36.875176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:36.875139    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"
Nov 02 09:51:37.018651 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.016692    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:51:37.019436 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.019412    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:51:37.020947 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.020679    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:51:37.213280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.213149    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d92016-a072-4478-9231-12996763776e" path="/var/lib/kubelet/pods/81d92016-a072-4478-9231-12996763776e/volumes"
Nov 02 09:51:37.224042 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:37.223950    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a524abe_238d_4fcf_a096_d07032f33da4.slice/crio-0930ef692f7fde73e17c9fb94a6a03bd6c568267d05ab8146e2a1ccb57bdd3f8 WatchSource:0}: Error finding container 0930ef692f7fde73e17c9fb94a6a03bd6c568267d05ab8146e2a1ccb57bdd3f8: Status 404 returned error can't find the container with id 0930ef692f7fde73e17c9fb94a6a03bd6c568267d05ab8146e2a1ccb57bdd3f8
Nov 02 09:51:37.225570 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.225542    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"]
Nov 02 09:51:37.268167 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.268134    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc" event={"ID":"6a524abe-238d-4fcf-a096-d07032f33da4","Type":"ContainerStarted","Data":"0930ef692f7fde73e17c9fb94a6a03bd6c568267d05ab8146e2a1ccb57bdd3f8"}
Nov 02 09:51:37.269628 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.269604    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"
Nov 02 09:51:37.269723 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.269608    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl" event={"ID":"c4674912-6f57-42d8-b948-a36fd412d1b7","Type":"ContainerDied","Data":"021daa025f183c0e00a2c07bcd41de209e60ccd210ecaa2cef7905c3aa8f9a7a"}
Nov 02 09:51:37.269723 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.269663    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="021daa025f183c0e00a2c07bcd41de209e60ccd210ecaa2cef7905c3aa8f9a7a"
Nov 02 09:51:37.270070 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.270052    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:37.270926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.270903    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:51:37.292378 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.292352    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"]
Nov 02 09:51:37.298070 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.298032    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-712810ea-sftp7"]
Nov 02 09:51:37.494438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.494407    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"]
Nov 02 09:51:37.533210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.533152    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs" podUID="616acf36-b808-4182-bfc6-55ec2849bc0c" containerName="payload-processing" containerID="cri-o://72b277a2fa5ba2fe67b54fc4ad13cc741419ff02cedb6d560e91c9f9620e6c06" gracePeriod=25
Nov 02 09:51:37.635262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.634015    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv" podUID="96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee" containerName="payload-pre-processing" containerID="cri-o://261902a77bcfe3f33718220b4a90113fb2ca393b23af483e28033eddd6e7a31d" gracePeriod=25
Nov 02 09:51:37.896755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.896727    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"
Nov 02 09:51:37.940857 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.940830    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"
Nov 02 09:51:37.975994 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.975962    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616acf36-b808-4182-bfc6-55ec2849bc0c-kube-api-access-984xn\" (UniqueName: \"kubernetes.io/projected/616acf36-b808-4182-bfc6-55ec2849bc0c-kube-api-access-984xn\") pod \"616acf36-b808-4182-bfc6-55ec2849bc0c\" (UID: \"616acf36-b808-4182-bfc6-55ec2849bc0c\") "
Nov 02 09:51:37.976137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.976069    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-kube-api-access-kx698\" (UniqueName: \"kubernetes.io/projected/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-kube-api-access-kx698\") pod \"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee\" (UID: \"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee\") "
Nov 02 09:51:37.976194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.976134    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-plugins-config-volume\") pod \"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee\" (UID: \"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee\") "
Nov 02 09:51:37.976284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.976210    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616acf36-b808-4182-bfc6-55ec2849bc0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616acf36-b808-4182-bfc6-55ec2849bc0c-plugins-config-volume\") pod \"616acf36-b808-4182-bfc6-55ec2849bc0c\" (UID: \"616acf36-b808-4182-bfc6-55ec2849bc0c\") "
Nov 02 09:51:37.976897 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.976777    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-plugins-config-volume" pod "96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee" (UID: "96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:37.976897 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.976856    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616acf36-b808-4182-bfc6-55ec2849bc0c-plugins-config-volume" pod "616acf36-b808-4182-bfc6-55ec2849bc0c" (UID: "616acf36-b808-4182-bfc6-55ec2849bc0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:37.977104 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.976991    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:37.977104 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.977012    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616acf36-b808-4182-bfc6-55ec2849bc0c-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:37.978600 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.978573    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-kube-api-access-kx698" pod "96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee" (UID: "96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee"). InnerVolumeSpecName "kube-api-access-kx698". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:37.979103 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:37.979082    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616acf36-b808-4182-bfc6-55ec2849bc0c-kube-api-access-984xn" pod "616acf36-b808-4182-bfc6-55ec2849bc0c" (UID: "616acf36-b808-4182-bfc6-55ec2849bc0c"). InnerVolumeSpecName "kube-api-access-984xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:38.078371 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.078346    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-984xn\" (UniqueName: \"kubernetes.io/projected/616acf36-b808-4182-bfc6-55ec2849bc0c-kube-api-access-984xn\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:38.078371 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.078370    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx698\" (UniqueName: \"kubernetes.io/projected/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee-kube-api-access-kx698\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:38.279829 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.279792    4449 generic.go:356] "Generic (PLEG): container finished" podID="96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee" containerID="261902a77bcfe3f33718220b4a90113fb2ca393b23af483e28033eddd6e7a31d" exitCode=0
Nov 02 09:51:38.280018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.279833    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv" event={"ID":"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee","Type":"ContainerDied","Data":"261902a77bcfe3f33718220b4a90113fb2ca393b23af483e28033eddd6e7a31d"}
Nov 02 09:51:38.280018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.279842    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"
Nov 02 09:51:38.280018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.279893    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv" event={"ID":"96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee","Type":"ContainerDied","Data":"5f90ee368e694f58d4828c3036983aaf2d8cfb28a2c0344ec814b3e3948b433d"}
Nov 02 09:51:38.280018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.279924    4449 scope.go:122] "RemoveContainer" containerID="261902a77bcfe3f33718220b4a90113fb2ca393b23af483e28033eddd6e7a31d"
Nov 02 09:51:38.281610 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.281586    4449 generic.go:356] "Generic (PLEG): container finished" podID="616acf36-b808-4182-bfc6-55ec2849bc0c" containerID="72b277a2fa5ba2fe67b54fc4ad13cc741419ff02cedb6d560e91c9f9620e6c06" exitCode=0
Nov 02 09:51:38.281735 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.281640    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"
Nov 02 09:51:38.281735 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.281670    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs" event={"ID":"616acf36-b808-4182-bfc6-55ec2849bc0c","Type":"ContainerDied","Data":"72b277a2fa5ba2fe67b54fc4ad13cc741419ff02cedb6d560e91c9f9620e6c06"}
Nov 02 09:51:38.281858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.281734    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs" event={"ID":"616acf36-b808-4182-bfc6-55ec2849bc0c","Type":"ContainerDied","Data":"510c4c185eabc452d007c055eb8a499a2e7689b86ed8934657a853c3f17389e8"}
Nov 02 09:51:38.283829 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.283805    4449 generic.go:356] "Generic (PLEG): container finished" podID="6a524abe-238d-4fcf-a096-d07032f33da4" containerID="5137a92025fdf2825257453777f61801e8133822a045fca0be4d8f9ddd667fdc" exitCode=0
Nov 02 09:51:38.283944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.283890    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc" event={"ID":"6a524abe-238d-4fcf-a096-d07032f33da4","Type":"ContainerDied","Data":"5137a92025fdf2825257453777f61801e8133822a045fca0be4d8f9ddd667fdc"}
Nov 02 09:51:38.284565 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.284542    4449 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" secret="" err="secret \"e2e-ait-712810ea-openshift-default-dockercfg-zbfsq\" not found"
Nov 02 09:51:38.293860 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.293844    4449 scope.go:122] "RemoveContainer" containerID="261902a77bcfe3f33718220b4a90113fb2ca393b23af483e28033eddd6e7a31d"
Nov 02 09:51:38.294111 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:51:38.294092    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261902a77bcfe3f33718220b4a90113fb2ca393b23af483e28033eddd6e7a31d\": container with ID starting with 261902a77bcfe3f33718220b4a90113fb2ca393b23af483e28033eddd6e7a31d not found: ID does not exist" containerID="261902a77bcfe3f33718220b4a90113fb2ca393b23af483e28033eddd6e7a31d"
Nov 02 09:51:38.294176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.294120    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261902a77bcfe3f33718220b4a90113fb2ca393b23af483e28033eddd6e7a31d"} err="failed to get container status \"261902a77bcfe3f33718220b4a90113fb2ca393b23af483e28033eddd6e7a31d\": rpc error: code = NotFound desc = could not find container \"261902a77bcfe3f33718220b4a90113fb2ca393b23af483e28033eddd6e7a31d\": container with ID starting with 261902a77bcfe3f33718220b4a90113fb2ca393b23af483e28033eddd6e7a31d not found: ID does not exist"
Nov 02 09:51:38.294176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.294142    4449 scope.go:122] "RemoveContainer" containerID="72b277a2fa5ba2fe67b54fc4ad13cc741419ff02cedb6d560e91c9f9620e6c06"
Nov 02 09:51:38.307939 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.307920    4449 scope.go:122] "RemoveContainer" containerID="72b277a2fa5ba2fe67b54fc4ad13cc741419ff02cedb6d560e91c9f9620e6c06"
Nov 02 09:51:38.308203 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:51:38.308179    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b277a2fa5ba2fe67b54fc4ad13cc741419ff02cedb6d560e91c9f9620e6c06\": container with ID starting with 72b277a2fa5ba2fe67b54fc4ad13cc741419ff02cedb6d560e91c9f9620e6c06 not found: ID does not exist" containerID="72b277a2fa5ba2fe67b54fc4ad13cc741419ff02cedb6d560e91c9f9620e6c06"
Nov 02 09:51:38.308303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.308229    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b277a2fa5ba2fe67b54fc4ad13cc741419ff02cedb6d560e91c9f9620e6c06"} err="failed to get container status \"72b277a2fa5ba2fe67b54fc4ad13cc741419ff02cedb6d560e91c9f9620e6c06\": rpc error: code = NotFound desc = could not find container \"72b277a2fa5ba2fe67b54fc4ad13cc741419ff02cedb6d560e91c9f9620e6c06\": container with ID starting with 72b277a2fa5ba2fe67b54fc4ad13cc741419ff02cedb6d560e91c9f9620e6c06 not found: ID does not exist"
Nov 02 09:51:38.367684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.367600    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"]
Nov 02 09:51:38.372717 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.372691    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-712810ea-8449cd864f-l84rs"]
Nov 02 09:51:38.387933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.387908    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"]
Nov 02 09:51:38.396272 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.396249    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-712810ea-77c968f8c9-4clwv"]
Nov 02 09:51:38.918007 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.917973    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"]
Nov 02 09:51:38.919191 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.919149    4449 cpu_manager.go:414] "containerMap: removing container" podUID="616acf36-b808-4182-bfc6-55ec2849bc0c" containerName="payload-processing"
Nov 02 09:51:38.919191 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.919179    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616acf36-b808-4182-bfc6-55ec2849bc0c" containerName="payload-processing"
Nov 02 09:51:38.919449 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.919250    4449 cpu_manager.go:414] "containerMap: removing container" podUID="96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee" containerName="payload-pre-processing"
Nov 02 09:51:38.919449 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.919259    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee" containerName="payload-pre-processing"
Nov 02 09:51:38.919449 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.919431    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="616acf36-b808-4182-bfc6-55ec2849bc0c" containerName="payload-processing"
Nov 02 09:51:38.919449 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.919453    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee" containerName="payload-pre-processing"
Nov 02 09:51:38.924006 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.923978    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:38.929643 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.929622    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4c342351-gw-openshift-default-dockercfg-gfvzn\""
Nov 02 09:51:38.936755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.936728    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"]
Nov 02 09:51:38.990286 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.990246    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qmd2\" (UniqueName: \"kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-kube-api-access-7qmd2\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:38.990500 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.990326    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-envoy\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:38.990500 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.990489    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-data\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:38.990701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.990675    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-token\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:38.990786 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.990720    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-socket\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:38.990833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.990821    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-certs\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:38.990877 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.990865    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-credential-socket\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:38.990984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.990962    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-podinfo\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:38.991017 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:38.990998    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istiod-ca-cert\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.091913 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.091875    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-podinfo\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.091913 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.091916    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istiod-ca-cert\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.092382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.091958    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qmd2\" (UniqueName: \"kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-kube-api-access-7qmd2\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.092382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.092085    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-envoy\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.092382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.092152    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-data\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.092382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.092310    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-token\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.092382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.092338    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-socket\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.092382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.092378    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-certs\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.092696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.092454    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-credential-socket\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.092696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.092562    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-data\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.092791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.092743    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-certs\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.092828 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.092804    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-credential-socket\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.092958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.092943    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-socket\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.092995 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.092971    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istiod-ca-cert\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.094558 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.094532    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-envoy\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.094648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.094596    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-podinfo\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.110945 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.110834    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-token\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.114881 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.114856    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qmd2\" (UniqueName: \"kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-kube-api-access-7qmd2\") pod \"e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") " pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.183850 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.183771    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"]
Nov 02 09:51:39.190071 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.190039    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"
Nov 02 09:51:39.196103 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.196080    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"]
Nov 02 09:51:39.199591 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.199568    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9cbd331c\""
Nov 02 09:51:39.204023 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.203997    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"
Nov 02 09:51:39.219959 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.219926    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9cbd331c-dockercfg-6dnlq\""
Nov 02 09:51:39.225938 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.225913    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33be085e-851a-46a2-b44c-524abb75390a" path="/var/lib/kubelet/pods/33be085e-851a-46a2-b44c-524abb75390a/volumes"
Nov 02 09:51:39.226540 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.226520    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="616acf36-b808-4182-bfc6-55ec2849bc0c" path="/var/lib/kubelet/pods/616acf36-b808-4182-bfc6-55ec2849bc0c/volumes"
Nov 02 09:51:39.227107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.227088    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee" path="/var/lib/kubelet/pods/96e1cfb2-c0f2-40e1-b1eb-22f2c14b2aee/volumes"
Nov 02 09:51:39.227577 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.227557    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"]
Nov 02 09:51:39.227648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.227618    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"]
Nov 02 09:51:39.242456 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.242431    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:39.294696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.294657    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74a3871-b77b-41e3-85b8-f7bbb4e01651-plugins-config-volume\") pod \"payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb\" (UID: \"b74a3871-b77b-41e3-85b8-f7bbb4e01651\") " pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"
Nov 02 09:51:39.294865 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.294766    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6\" (UID: \"0a77eafb-2709-47c6-870e-4fd9f7da6e6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"
Nov 02 09:51:39.294865 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.294833    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnnnj\" (UniqueName: \"kubernetes.io/projected/b74a3871-b77b-41e3-85b8-f7bbb4e01651-kube-api-access-lnnnj\") pod \"payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb\" (UID: \"b74a3871-b77b-41e3-85b8-f7bbb4e01651\") " pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"
Nov 02 09:51:39.294985 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.294934    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhp45\" (UniqueName: \"kubernetes.io/projected/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-kube-api-access-dhp45\") pod \"payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6\" (UID: \"0a77eafb-2709-47c6-870e-4fd9f7da6e6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"
Nov 02 09:51:39.296904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.296872    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy" containerID="cri-o://609c052c784857f1319247abe88a9263dd3abba05768b2ade2439c78f0e0edfd" gracePeriod=30
Nov 02 09:51:39.438054 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.401170    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74a3871-b77b-41e3-85b8-f7bbb4e01651-plugins-config-volume\") pod \"payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb\" (UID: \"b74a3871-b77b-41e3-85b8-f7bbb4e01651\") " pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"
Nov 02 09:51:39.438054 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.401268    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6\" (UID: \"0a77eafb-2709-47c6-870e-4fd9f7da6e6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"
Nov 02 09:51:39.438054 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.401329    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnnnj\" (UniqueName: \"kubernetes.io/projected/b74a3871-b77b-41e3-85b8-f7bbb4e01651-kube-api-access-lnnnj\") pod \"payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb\" (UID: \"b74a3871-b77b-41e3-85b8-f7bbb4e01651\") " pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"
Nov 02 09:51:39.438054 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.401403    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhp45\" (UniqueName: \"kubernetes.io/projected/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-kube-api-access-dhp45\") pod \"payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6\" (UID: \"0a77eafb-2709-47c6-870e-4fd9f7da6e6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"
Nov 02 09:51:39.438054 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.410654    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6\" (UID: \"0a77eafb-2709-47c6-870e-4fd9f7da6e6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"
Nov 02 09:51:39.438054 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.411916    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74a3871-b77b-41e3-85b8-f7bbb4e01651-plugins-config-volume\") pod \"payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb\" (UID: \"b74a3871-b77b-41e3-85b8-f7bbb4e01651\") " pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"
Nov 02 09:51:39.439539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.439475    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhp45\" (UniqueName: \"kubernetes.io/projected/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-kube-api-access-dhp45\") pod \"payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6\" (UID: \"0a77eafb-2709-47c6-870e-4fd9f7da6e6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"
Nov 02 09:51:39.453929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.453897    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnnnj\" (UniqueName: \"kubernetes.io/projected/b74a3871-b77b-41e3-85b8-f7bbb4e01651-kube-api-access-lnnnj\") pod \"payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb\" (UID: \"b74a3871-b77b-41e3-85b8-f7bbb4e01651\") " pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"
Nov 02 09:51:39.510634 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.510596    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"
Nov 02 09:51:39.534473 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.534449    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"
Nov 02 09:51:39.653868 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.653640    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"
Nov 02 09:51:39.748156 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:39.748117    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1243f9d4_8d6f_4ab0_8a86_bd5ccbf2b679.slice/crio-e5e494450f4149d332ec4e653ada89c47df1dbdb9dec4f59e14957ffef952894 WatchSource:0}: Error finding container e5e494450f4149d332ec4e653ada89c47df1dbdb9dec4f59e14957ffef952894: Status 404 returned error can't find the container with id e5e494450f4149d332ec4e653ada89c47df1dbdb9dec4f59e14957ffef952894
Nov 02 09:51:39.750066 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.750043    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"]
Nov 02 09:51:39.751155 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.751118    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:51:39.751261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.751201    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:51:39.751334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.751263    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:51:39.809782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.809755    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-openshift-ca\") pod \"6a524abe-238d-4fcf-a096-d07032f33da4\" (UID: \"6a524abe-238d-4fcf-a096-d07032f33da4\") "
Nov 02 09:51:39.809952 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.809913    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-xks-ca\") pod \"6a524abe-238d-4fcf-a096-d07032f33da4\" (UID: \"6a524abe-238d-4fcf-a096-d07032f33da4\") "
Nov 02 09:51:39.810018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.809952    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a524abe-238d-4fcf-a096-d07032f33da4-kube-api-access-gmgkv\" (UniqueName: \"kubernetes.io/projected/6a524abe-238d-4fcf-a096-d07032f33da4-kube-api-access-gmgkv\") pod \"6a524abe-238d-4fcf-a096-d07032f33da4\" (UID: \"6a524abe-238d-4fcf-a096-d07032f33da4\") "
Nov 02 09:51:39.812493 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.812404    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-openshift-ca" pod "6a524abe-238d-4fcf-a096-d07032f33da4" (UID: "6a524abe-238d-4fcf-a096-d07032f33da4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:39.812971 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.812944    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a524abe-238d-4fcf-a096-d07032f33da4-kube-api-access-gmgkv" pod "6a524abe-238d-4fcf-a096-d07032f33da4" (UID: "6a524abe-238d-4fcf-a096-d07032f33da4"). InnerVolumeSpecName "kube-api-access-gmgkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:39.814679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.814651    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-xks-ca" pod "6a524abe-238d-4fcf-a096-d07032f33da4" (UID: "6a524abe-238d-4fcf-a096-d07032f33da4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:51:39.816810 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:39.816784    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74a3871_b77b_41e3_85b8_f7bbb4e01651.slice/crio-526f3d59e8c54ec44eadf49293334c40d218a8bf84f4127cbe7ed91c10f133f9 WatchSource:0}: Error finding container 526f3d59e8c54ec44eadf49293334c40d218a8bf84f4127cbe7ed91c10f133f9: Status 404 returned error can't find the container with id 526f3d59e8c54ec44eadf49293334c40d218a8bf84f4127cbe7ed91c10f133f9
Nov 02 09:51:39.816999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.816979    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"]
Nov 02 09:51:39.860677 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.860392    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"]
Nov 02 09:51:39.911963 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.911931    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:39.912093 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.911967    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a524abe-238d-4fcf-a096-d07032f33da4-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:39.912093 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:39.911986    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmgkv\" (UniqueName: \"kubernetes.io/projected/6a524abe-238d-4fcf-a096-d07032f33da4-kube-api-access-gmgkv\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:40.309290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.309249    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj" event={"ID":"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679","Type":"ContainerStarted","Data":"e9926a181c02240a8d90a9e33ed86299ad4920c90f7af04aae7913a7a427c360"}
Nov 02 09:51:40.309745 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.309307    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj" event={"ID":"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679","Type":"ContainerStarted","Data":"e5e494450f4149d332ec4e653ada89c47df1dbdb9dec4f59e14957ffef952894"}
Nov 02 09:51:40.310942 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.310918    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"
Nov 02 09:51:40.311079 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.310954    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc" event={"ID":"6a524abe-238d-4fcf-a096-d07032f33da4","Type":"ContainerDied","Data":"0930ef692f7fde73e17c9fb94a6a03bd6c568267d05ab8146e2a1ccb57bdd3f8"}
Nov 02 09:51:40.311079 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.311002    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0930ef692f7fde73e17c9fb94a6a03bd6c568267d05ab8146e2a1ccb57bdd3f8"
Nov 02 09:51:40.315405 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.315377    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb" event={"ID":"b74a3871-b77b-41e3-85b8-f7bbb4e01651","Type":"ContainerStarted","Data":"472bdace349c3ee8b68e7db7efbb255ef70435737d639af7e23f8e178a6de335"}
Nov 02 09:51:40.315596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.315431    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb" event={"ID":"b74a3871-b77b-41e3-85b8-f7bbb4e01651","Type":"ContainerStarted","Data":"526f3d59e8c54ec44eadf49293334c40d218a8bf84f4127cbe7ed91c10f133f9"}
Nov 02 09:51:40.315596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.315490    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"
Nov 02 09:51:40.317196 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.317171    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6" event={"ID":"0a77eafb-2709-47c6-870e-4fd9f7da6e6f","Type":"ContainerStarted","Data":"1ce9454695986e09fb6669e57518d6093747c05a39e93a4f0b799dd293e18e2d"}
Nov 02 09:51:40.317329 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.317255    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6" event={"ID":"0a77eafb-2709-47c6-870e-4fd9f7da6e6f","Type":"ContainerStarted","Data":"c259e3fa03c6476a1c42495f0c7ba42392cb75566e04e9b535982393f49f3401"}
Nov 02 09:51:40.317389 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.317336    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"
Nov 02 09:51:40.340744 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.340694    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj" podStartSLOduration=2.340678536 podStartE2EDuration="2.340678536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:40.335341848 +0000 UTC m=+977.273628625" watchObservedRunningTime="2026-11-02 09:51:40.340678536 +0000 UTC m=+977.278965308"
Nov 02 09:51:40.360369 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.360315    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6" podStartSLOduration=1.3602973440000001 podStartE2EDuration="1.360297344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:40.358066103 +0000 UTC m=+977.296352902" watchObservedRunningTime="2026-11-02 09:51:40.360297344 +0000 UTC m=+977.298584120"
Nov 02 09:51:40.391301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:40.391243    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb" podStartSLOduration=1.391206461 podStartE2EDuration="1.391206461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:40.383418292 +0000 UTC m=+977.321705065" watchObservedRunningTime="2026-11-02 09:51:40.391206461 +0000 UTC m=+977.329493239"
Nov 02 09:51:41.253659 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:41.253623    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:41.266572 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:41.266539    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:41.393247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:41.392252    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:41.410255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:41.395431    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:51:42.042617 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:42.042561    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"]
Nov 02 09:51:42.048194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:42.048166    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c06d57b-7xsl4"]
Nov 02 09:51:43.209873 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.209840    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48777f59-1ff1-4d26-a615-22c1f9ee1ff7" path="/var/lib/kubelet/pods/48777f59-1ff1-4d26-a615-22c1f9ee1ff7/volumes"
Nov 02 09:51:43.266373 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.266336    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"]
Nov 02 09:51:43.267005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.266992    4449 cpu_manager.go:414] "containerMap: removing container" podUID="6a524abe-238d-4fcf-a096-d07032f33da4" containerName="revoke-keys"
Nov 02 09:51:43.267048 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.267007    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a524abe-238d-4fcf-a096-d07032f33da4" containerName="revoke-keys"
Nov 02 09:51:43.267159 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.267149    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a524abe-238d-4fcf-a096-d07032f33da4" containerName="revoke-keys"
Nov 02 09:51:43.272399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.272374    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"
Nov 02 09:51:43.282485 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.282457    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4c342351-dockercfg-zd9gm\""
Nov 02 09:51:43.282485 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.282479    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4c342351\""
Nov 02 09:51:43.286188 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.286166    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"]
Nov 02 09:51:43.357679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.357637    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cpcw\" (UniqueName: \"kubernetes.io/projected/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-kube-api-access-8cpcw\") pod \"payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859\" (UID: \"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"
Nov 02 09:51:43.357872 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.357842    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859\" (UID: \"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"
Nov 02 09:51:43.512451 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.460188    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859\" (UID: \"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"
Nov 02 09:51:43.512679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.512613    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cpcw\" (UniqueName: \"kubernetes.io/projected/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-kube-api-access-8cpcw\") pod \"payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859\" (UID: \"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"
Nov 02 09:51:43.513234 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.478359    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859\" (UID: \"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"
Nov 02 09:51:43.528109 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.528063    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cpcw\" (UniqueName: \"kubernetes.io/projected/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-kube-api-access-8cpcw\") pod \"payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859\" (UID: \"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"
Nov 02 09:51:43.590601 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.590568    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"
Nov 02 09:51:43.874413 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:43.874379    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67b2b4d_d0b9_4245_9fba_6b8a9671eb9b.slice/crio-07016e0b2c9e876579f336876a54125ee2712a5d502c44aac4ea7ff3d31b742b WatchSource:0}: Error finding container 07016e0b2c9e876579f336876a54125ee2712a5d502c44aac4ea7ff3d31b742b: Status 404 returned error can't find the container with id 07016e0b2c9e876579f336876a54125ee2712a5d502c44aac4ea7ff3d31b742b
Nov 02 09:51:43.877884 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:43.877852    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"]
Nov 02 09:51:44.231998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.231897    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:51:44.294524 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.293338    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"]
Nov 02 09:51:44.528146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.528049    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859" event={"ID":"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b","Type":"ContainerStarted","Data":"2f96fff47ffc92d9e231b808597b71f98ac2fde6243044680db6386dcc7b3c62"}
Nov 02 09:51:44.528146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.528118    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859" event={"ID":"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b","Type":"ContainerStarted","Data":"07016e0b2c9e876579f336876a54125ee2712a5d502c44aac4ea7ff3d31b742b"}
Nov 02 09:51:44.528484 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.528463    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"
Nov 02 09:51:44.530602 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.530577    4449 generic.go:356] "Generic (PLEG): container finished" podID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerID="609c052c784857f1319247abe88a9263dd3abba05768b2ade2439c78f0e0edfd" exitCode=0
Nov 02 09:51:44.530809 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.530793    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" event={"ID":"22f0ca6c-1150-4c10-9510-3613a96e425c","Type":"ContainerDied","Data":"609c052c784857f1319247abe88a9263dd3abba05768b2ade2439c78f0e0edfd"}
Nov 02 09:51:44.530872 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.530845    4449 scope.go:122] "RemoveContainer" containerID="2f27071bfe5e8042abf5139c2229932e2326959715a89dec6a852cc70040987a"
Nov 02 09:51:44.549651 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.549584    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859" podStartSLOduration=1.5495647209999999 podStartE2EDuration="1.549564721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:44.546502856 +0000 UTC m=+981.484789633" watchObservedRunningTime="2026-11-02 09:51:44.549564721 +0000 UTC m=+981.487851496"
Nov 02 09:51:44.593623 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.593592    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:51:44.730081 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.730045    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f0ca6c-1150-4c10-9510-3613a96e425c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f0ca6c-1150-4c10-9510-3613a96e425c-istiod-ca-cert\") pod \"22f0ca6c-1150-4c10-9510-3613a96e425c\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") "
Nov 02 09:51:44.730081 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.730085    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-podinfo\") pod \"22f0ca6c-1150-4c10-9510-3613a96e425c\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") "
Nov 02 09:51:44.730374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.730144    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-socket\") pod \"22f0ca6c-1150-4c10-9510-3613a96e425c\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") "
Nov 02 09:51:44.730374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.730195    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-data\") pod \"22f0ca6c-1150-4c10-9510-3613a96e425c\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") "
Nov 02 09:51:44.730374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.730231    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-envoy\") pod \"22f0ca6c-1150-4c10-9510-3613a96e425c\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") "
Nov 02 09:51:44.730374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.730270    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-token\" (UniqueName: \"kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-token\") pod \"22f0ca6c-1150-4c10-9510-3613a96e425c\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") "
Nov 02 09:51:44.730374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.730327    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-kube-api-access-wqlqf\" (UniqueName: \"kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-kube-api-access-wqlqf\") pod \"22f0ca6c-1150-4c10-9510-3613a96e425c\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") "
Nov 02 09:51:44.730619 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.730406    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-credential-socket\") pod \"22f0ca6c-1150-4c10-9510-3613a96e425c\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") "
Nov 02 09:51:44.730619 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.730423    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-certs\") pod \"22f0ca6c-1150-4c10-9510-3613a96e425c\" (UID: \"22f0ca6c-1150-4c10-9510-3613a96e425c\") "
Nov 02 09:51:44.731030 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.730999    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-data" pod "22f0ca6c-1150-4c10-9510-3613a96e425c" (UID: "22f0ca6c-1150-4c10-9510-3613a96e425c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:51:44.731173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.731007    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-certs" pod "22f0ca6c-1150-4c10-9510-3613a96e425c" (UID: "22f0ca6c-1150-4c10-9510-3613a96e425c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:51:44.731533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.731501    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-credential-socket" pod "22f0ca6c-1150-4c10-9510-3613a96e425c" (UID: "22f0ca6c-1150-4c10-9510-3613a96e425c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:51:44.731778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.731758    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f0ca6c-1150-4c10-9510-3613a96e425c-istiod-ca-cert" pod "22f0ca6c-1150-4c10-9510-3613a96e425c" (UID: "22f0ca6c-1150-4c10-9510-3613a96e425c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:44.732003 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.731979    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-socket" pod "22f0ca6c-1150-4c10-9510-3613a96e425c" (UID: "22f0ca6c-1150-4c10-9510-3613a96e425c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:51:44.734530 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.734502    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-kube-api-access-wqlqf" pod "22f0ca6c-1150-4c10-9510-3613a96e425c" (UID: "22f0ca6c-1150-4c10-9510-3613a96e425c"). InnerVolumeSpecName "kube-api-access-wqlqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:44.734530 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.734512    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-envoy" pod "22f0ca6c-1150-4c10-9510-3613a96e425c" (UID: "22f0ca6c-1150-4c10-9510-3613a96e425c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:51:44.734679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.734535    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-token" pod "22f0ca6c-1150-4c10-9510-3613a96e425c" (UID: "22f0ca6c-1150-4c10-9510-3613a96e425c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:44.735787 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.735767    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-podinfo" pod "22f0ca6c-1150-4c10-9510-3613a96e425c" (UID: "22f0ca6c-1150-4c10-9510-3613a96e425c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:51:44.831729 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.831698    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:44.831729 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.831726    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:44.831729 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.831735    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:44.831988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.831745    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqlqf\" (UniqueName: \"kubernetes.io/projected/22f0ca6c-1150-4c10-9510-3613a96e425c-kube-api-access-wqlqf\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:44.831988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.831756    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:44.831988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.831764    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:44.831988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.831772    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f0ca6c-1150-4c10-9510-3613a96e425c-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:44.831988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.831780    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f0ca6c-1150-4c10-9510-3613a96e425c-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:44.831988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:44.831788    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f0ca6c-1150-4c10-9510-3613a96e425c-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:45.540383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:45.540353    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"
Nov 02 09:51:45.540878 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:45.540382    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6" event={"ID":"22f0ca6c-1150-4c10-9510-3613a96e425c","Type":"ContainerDied","Data":"8e63b4eb4b083f60fd73d6b84686a561d737701462f79255763b676bcb66c46c"}
Nov 02 09:51:45.540878 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:45.540452    4449 scope.go:122] "RemoveContainer" containerID="609c052c784857f1319247abe88a9263dd3abba05768b2ade2439c78f0e0edfd"
Nov 02 09:51:45.564701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:45.564667    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"]
Nov 02 09:51:45.570162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:45.570139    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-712810ea-openshift-default-76c9fb4567-ncpx6"]
Nov 02 09:51:46.305804 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.305774    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"]
Nov 02 09:51:46.306443 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.306429    4449 cpu_manager.go:414] "containerMap: removing container" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy"
Nov 02 09:51:46.306502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.306444    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy"
Nov 02 09:51:46.306502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.306464    4449 cpu_manager.go:414] "containerMap: removing container" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy"
Nov 02 09:51:46.306502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.306470    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy"
Nov 02 09:51:46.306599 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.306589    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy"
Nov 02 09:51:46.306632 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.306603    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" containerName="istio-proxy"
Nov 02 09:51:46.311772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.311750    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"
Nov 02 09:51:46.318258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.318187    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:51:46.318933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.318620    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:51:46.318933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.318687    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:51:46.318933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.318730    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:51:46.320776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.320752    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"]
Nov 02 09:51:46.453570 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.453534    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjppb\" (UniqueName: \"kubernetes.io/projected/dc77c739-002f-4c2a-bdc4-e68ef8472812-kube-api-access-qjppb\") pod \"maas-api-revoke-sub-75d1d900-2vhcp\" (UID: \"dc77c739-002f-4c2a-bdc4-e68ef8472812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"
Nov 02 09:51:46.453811 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.453722    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d1d900-2vhcp\" (UID: \"dc77c739-002f-4c2a-bdc4-e68ef8472812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"
Nov 02 09:51:46.453811 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.453779    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d1d900-2vhcp\" (UID: \"dc77c739-002f-4c2a-bdc4-e68ef8472812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"
Nov 02 09:51:46.555039 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.555007    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjppb\" (UniqueName: \"kubernetes.io/projected/dc77c739-002f-4c2a-bdc4-e68ef8472812-kube-api-access-qjppb\") pod \"maas-api-revoke-sub-75d1d900-2vhcp\" (UID: \"dc77c739-002f-4c2a-bdc4-e68ef8472812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"
Nov 02 09:51:46.555545 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.555183    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d1d900-2vhcp\" (UID: \"dc77c739-002f-4c2a-bdc4-e68ef8472812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"
Nov 02 09:51:46.555545 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.555277    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d1d900-2vhcp\" (UID: \"dc77c739-002f-4c2a-bdc4-e68ef8472812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"
Nov 02 09:51:46.555858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.555793    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d1d900-2vhcp\" (UID: \"dc77c739-002f-4c2a-bdc4-e68ef8472812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"
Nov 02 09:51:46.557927 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.557849    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d1d900-2vhcp\" (UID: \"dc77c739-002f-4c2a-bdc4-e68ef8472812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"
Nov 02 09:51:46.568379 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.568350    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjppb\" (UniqueName: \"kubernetes.io/projected/dc77c739-002f-4c2a-bdc4-e68ef8472812-kube-api-access-qjppb\") pod \"maas-api-revoke-sub-75d1d900-2vhcp\" (UID: \"dc77c739-002f-4c2a-bdc4-e68ef8472812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"
Nov 02 09:51:46.630637 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.630592    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"
Nov 02 09:51:46.916577 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:46.916545    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc77c739_002f_4c2a_bdc4_e68ef8472812.slice/crio-ea1d47a0d88f7b3df0f6240be9a0a06c08aaa86d8a0b75d501a7ee0a35eeb836 WatchSource:0}: Error finding container ea1d47a0d88f7b3df0f6240be9a0a06c08aaa86d8a0b75d501a7ee0a35eeb836: Status 404 returned error can't find the container with id ea1d47a0d88f7b3df0f6240be9a0a06c08aaa86d8a0b75d501a7ee0a35eeb836
Nov 02 09:51:46.917862 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:46.917839    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"]
Nov 02 09:51:47.155881 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:47.155654    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"]
Nov 02 09:51:47.208440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:47.208357    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f0ca6c-1150-4c10-9510-3613a96e425c" path="/var/lib/kubelet/pods/22f0ca6c-1150-4c10-9510-3613a96e425c/volumes"
Nov 02 09:51:47.565679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:47.565631    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp" event={"ID":"dc77c739-002f-4c2a-bdc4-e68ef8472812","Type":"ContainerStarted","Data":"d62673821730df0adf7ea1e4158b760f9b34fb0e65b3f9a4ce35983bbd183bf2"}
Nov 02 09:51:47.566122 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:47.565700    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp" event={"ID":"dc77c739-002f-4c2a-bdc4-e68ef8472812","Type":"ContainerStarted","Data":"ea1d47a0d88f7b3df0f6240be9a0a06c08aaa86d8a0b75d501a7ee0a35eeb836"}
Nov 02 09:51:47.585502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:47.585444    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp" podStartSLOduration=1.5854245740000001 podStartE2EDuration="1.585424574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:47.583626552 +0000 UTC m=+984.521913346" watchObservedRunningTime="2026-11-02 09:51:47.585424574 +0000 UTC m=+984.523711351"
Nov 02 09:51:48.054822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:48.054753    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"]
Nov 02 09:51:48.576317 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:48.576281    4449 generic.go:356] "Generic (PLEG): container finished" podID="dc77c739-002f-4c2a-bdc4-e68ef8472812" containerID="d62673821730df0adf7ea1e4158b760f9b34fb0e65b3f9a4ce35983bbd183bf2" exitCode=0
Nov 02 09:51:48.576769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:48.576376    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp" event={"ID":"dc77c739-002f-4c2a-bdc4-e68ef8472812","Type":"ContainerDied","Data":"d62673821730df0adf7ea1e4158b760f9b34fb0e65b3f9a4ce35983bbd183bf2"}
Nov 02 09:51:49.294046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.293998    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g" podUID="34535c8c-9a37-4955-ab9e-da3d09bf0d2d" containerName="payload-processing" containerID="cri-o://48f3510d90f49622654bf57904da427c6f86488f5c62ae32dbda6d8125e7c3ae" gracePeriod=25
Nov 02 09:51:49.572551 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.572525    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:51:49.586049 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.586025    4449 generic.go:356] "Generic (PLEG): container finished" podID="34535c8c-9a37-4955-ab9e-da3d09bf0d2d" containerID="48f3510d90f49622654bf57904da427c6f86488f5c62ae32dbda6d8125e7c3ae" exitCode=0
Nov 02 09:51:49.586492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.586067    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"
Nov 02 09:51:49.586492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.586111    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g" event={"ID":"34535c8c-9a37-4955-ab9e-da3d09bf0d2d","Type":"ContainerDied","Data":"48f3510d90f49622654bf57904da427c6f86488f5c62ae32dbda6d8125e7c3ae"}
Nov 02 09:51:49.586492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.586173    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g" event={"ID":"34535c8c-9a37-4955-ab9e-da3d09bf0d2d","Type":"ContainerDied","Data":"c9416ded70f123c1e3b67d239c9be8703a51d8f15364692c2015f7978b8af2fc"}
Nov 02 09:51:49.586492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.586207    4449 scope.go:122] "RemoveContainer" containerID="48f3510d90f49622654bf57904da427c6f86488f5c62ae32dbda6d8125e7c3ae"
Nov 02 09:51:49.600549 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.600526    4449 scope.go:122] "RemoveContainer" containerID="48f3510d90f49622654bf57904da427c6f86488f5c62ae32dbda6d8125e7c3ae"
Nov 02 09:51:49.600993 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:51:49.600966    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f3510d90f49622654bf57904da427c6f86488f5c62ae32dbda6d8125e7c3ae\": container with ID starting with 48f3510d90f49622654bf57904da427c6f86488f5c62ae32dbda6d8125e7c3ae not found: ID does not exist" containerID="48f3510d90f49622654bf57904da427c6f86488f5c62ae32dbda6d8125e7c3ae"
Nov 02 09:51:49.601099 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.601009    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f3510d90f49622654bf57904da427c6f86488f5c62ae32dbda6d8125e7c3ae"} err="failed to get container status \"48f3510d90f49622654bf57904da427c6f86488f5c62ae32dbda6d8125e7c3ae\": rpc error: code = NotFound desc = could not find container \"48f3510d90f49622654bf57904da427c6f86488f5c62ae32dbda6d8125e7c3ae\": container with ID starting with 48f3510d90f49622654bf57904da427c6f86488f5c62ae32dbda6d8125e7c3ae not found: ID does not exist"
Nov 02 09:51:49.718904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.718872    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-routing-overlay-volume\") pod \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") "
Nov 02 09:51:49.719075 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.719046    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-kube-api-access-25wj9\" (UniqueName: \"kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-kube-api-access-25wj9\") pod \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") "
Nov 02 09:51:49.719075 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.719070    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-provider-credentials\") pod \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") "
Nov 02 09:51:49.719205 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.719110    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-plugins-config-volume\") pod \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\" (UID: \"34535c8c-9a37-4955-ab9e-da3d09bf0d2d\") "
Nov 02 09:51:49.719327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.719302    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-routing-overlay-volume" pod "34535c8c-9a37-4955-ab9e-da3d09bf0d2d" (UID: "34535c8c-9a37-4955-ab9e-da3d09bf0d2d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:49.719579 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.719558    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-plugins-config-volume" pod "34535c8c-9a37-4955-ab9e-da3d09bf0d2d" (UID: "34535c8c-9a37-4955-ab9e-da3d09bf0d2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:49.719855 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.719834    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:49.719936 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.719861    4449 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-routing-overlay-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:49.721585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.721562    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-provider-credentials" pod "34535c8c-9a37-4955-ab9e-da3d09bf0d2d" (UID: "34535c8c-9a37-4955-ab9e-da3d09bf0d2d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:49.721675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.721596    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-kube-api-access-25wj9" pod "34535c8c-9a37-4955-ab9e-da3d09bf0d2d" (UID: "34535c8c-9a37-4955-ab9e-da3d09bf0d2d"). InnerVolumeSpecName "kube-api-access-25wj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:49.756141 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.756116    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"
Nov 02 09:51:49.821650 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.821555    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25wj9\" (UniqueName: \"kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-kube-api-access-25wj9\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:49.821650 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.821594    4449 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34535c8c-9a37-4955-ab9e-da3d09bf0d2d-provider-credentials\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:49.913053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.913023    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"]
Nov 02 09:51:49.918377 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.918355    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db558s9g"]
Nov 02 09:51:49.922895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.922872    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-openshift-ca\") pod \"dc77c739-002f-4c2a-bdc4-e68ef8472812\" (UID: \"dc77c739-002f-4c2a-bdc4-e68ef8472812\") "
Nov 02 09:51:49.923000 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.922951    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-xks-ca\") pod \"dc77c739-002f-4c2a-bdc4-e68ef8472812\" (UID: \"dc77c739-002f-4c2a-bdc4-e68ef8472812\") "
Nov 02 09:51:49.923109 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.923095    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc77c739-002f-4c2a-bdc4-e68ef8472812-kube-api-access-qjppb\" (UniqueName: \"kubernetes.io/projected/dc77c739-002f-4c2a-bdc4-e68ef8472812-kube-api-access-qjppb\") pod \"dc77c739-002f-4c2a-bdc4-e68ef8472812\" (UID: \"dc77c739-002f-4c2a-bdc4-e68ef8472812\") "
Nov 02 09:51:49.923328 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.923288    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-openshift-ca" pod "dc77c739-002f-4c2a-bdc4-e68ef8472812" (UID: "dc77c739-002f-4c2a-bdc4-e68ef8472812"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:49.923761 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.923710    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:49.925173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.925151    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-xks-ca" pod "dc77c739-002f-4c2a-bdc4-e68ef8472812" (UID: "dc77c739-002f-4c2a-bdc4-e68ef8472812"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:51:49.925308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:49.925291    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc77c739-002f-4c2a-bdc4-e68ef8472812-kube-api-access-qjppb" pod "dc77c739-002f-4c2a-bdc4-e68ef8472812" (UID: "dc77c739-002f-4c2a-bdc4-e68ef8472812"). InnerVolumeSpecName "kube-api-access-qjppb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:50.025191 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:50.025154    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjppb\" (UniqueName: \"kubernetes.io/projected/dc77c739-002f-4c2a-bdc4-e68ef8472812-kube-api-access-qjppb\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:50.025191 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:50.025183    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc77c739-002f-4c2a-bdc4-e68ef8472812-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:50.597556 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:50.597484    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"
Nov 02 09:51:50.597556 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:50.597521    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp" event={"ID":"dc77c739-002f-4c2a-bdc4-e68ef8472812","Type":"ContainerDied","Data":"ea1d47a0d88f7b3df0f6240be9a0a06c08aaa86d8a0b75d501a7ee0a35eeb836"}
Nov 02 09:51:50.597990 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:50.597572    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea1d47a0d88f7b3df0f6240be9a0a06c08aaa86d8a0b75d501a7ee0a35eeb836"
Nov 02 09:51:51.207946 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:51.207911    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34535c8c-9a37-4955-ab9e-da3d09bf0d2d" path="/var/lib/kubelet/pods/34535c8c-9a37-4955-ab9e-da3d09bf0d2d/volumes"
Nov 02 09:51:51.394596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:51.394562    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"
Nov 02 09:51:51.394890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:51.394856    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"
Nov 02 09:51:54.235158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.192305    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"]
Nov 02 09:51:54.262755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.262702    4449 cpu_manager.go:414] "containerMap: removing container" podUID="34535c8c-9a37-4955-ab9e-da3d09bf0d2d" containerName="payload-processing"
Nov 02 09:51:54.262755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.262752    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34535c8c-9a37-4955-ab9e-da3d09bf0d2d" containerName="payload-processing"
Nov 02 09:51:54.262988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.262784    4449 cpu_manager.go:414] "containerMap: removing container" podUID="dc77c739-002f-4c2a-bdc4-e68ef8472812" containerName="revoke-keys"
Nov 02 09:51:54.262988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.262813    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc77c739-002f-4c2a-bdc4-e68ef8472812" containerName="revoke-keys"
Nov 02 09:51:54.284740 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.284694    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="34535c8c-9a37-4955-ab9e-da3d09bf0d2d" containerName="payload-processing"
Nov 02 09:51:54.292581 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.292340    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc77c739-002f-4c2a-bdc4-e68ef8472812" containerName="revoke-keys"
Nov 02 09:51:54.366890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.366861    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.403640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.379290    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-bde944\"/\"sub-test-model-bde944-kserve-self-signed-certs\""
Nov 02 09:51:54.481901 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.481853    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"]
Nov 02 09:51:54.585305 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.514493    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-model-cache\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.611430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.611391    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tmp-dir\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.617457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.617426    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqwkf\" (UniqueName: \"kubernetes.io/projected/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-kube-api-access-rqwkf\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.642367 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.642334    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-dshm\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.642671 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.642650    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tls-certs\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.642833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.642817    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-home\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.746054 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.746016    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tls-certs\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.749104 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.749080    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-home\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.749719 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.749614    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-home\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.780964 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.780933    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tls-certs\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.781428 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.781402    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-model-cache\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.781949 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.781843    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-model-cache\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.782158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.782139    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tmp-dir\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.782654 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.782548    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tmp-dir\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.782911 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.782889    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqwkf\" (UniqueName: \"kubernetes.io/projected/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-kube-api-access-rqwkf\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.783538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.783518    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-dshm\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.795527 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.795500    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-dshm\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.831863 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.831832    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqwkf\" (UniqueName: \"kubernetes.io/projected/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-kube-api-access-rqwkf\") pod \"sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") " pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:54.907411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:54.907304    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:55.034517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:55.034452    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:51:55.525400 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:55.515306    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:55.585255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:55.577421    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"
Nov 02 09:51:55.910267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:51:55.910230    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4cec0e5_a1f2_4ebe_b72b_5ef446cfdeb9.slice/crio-aa607e8fc017f3afa53f3cecf10dc67d5ba357c9d6975326b56befca2891f556 WatchSource:0}: Error finding container aa607e8fc017f3afa53f3cecf10dc67d5ba357c9d6975326b56befca2891f556: Status 404 returned error can't find the container with id aa607e8fc017f3afa53f3cecf10dc67d5ba357c9d6975326b56befca2891f556
Nov 02 09:51:55.913643 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:55.913615    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"]
Nov 02 09:51:56.796285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:56.796250    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" event={"ID":"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9","Type":"ContainerStarted","Data":"13f9ec5a3c50fd979f836b85b296544e5a668ca60150b4ed711cf5e3e0fed402"}
Nov 02 09:51:56.796802 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:56.796305    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" event={"ID":"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9","Type":"ContainerStarted","Data":"aa607e8fc017f3afa53f3cecf10dc67d5ba357c9d6975326b56befca2891f556"}
Nov 02 09:51:56.818858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:56.818811    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" podStartSLOduration=3.818798756 podStartE2EDuration="3.818798756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:56.817435274 +0000 UTC m=+993.755722069" watchObservedRunningTime="2026-11-02 09:51:56.818798756 +0000 UTC m=+993.757085577"
Nov 02 09:51:57.035033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:51:57.035000    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:52:00.150753 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.150713    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"]
Nov 02 09:52:00.156613 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.156582    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.166168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.166104    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9cbd331c\"/\"openshift-service-ca.crt\""
Nov 02 09:52:00.166611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.166329    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9cbd331c\"/\"default-dockercfg-k84vz\""
Nov 02 09:52:00.166611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.166399    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9cbd331c\"/\"e2e-lifecycle-model-9cbd331c-kserve-self-signed-certs\""
Nov 02 09:52:00.166611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.166447    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9cbd331c\"/\"kube-root-ca.crt\""
Nov 02 09:52:00.177358 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.177321    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"]
Nov 02 09:52:00.212740 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.212629    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tchtz\" (UniqueName: \"kubernetes.io/projected/4d268d7a-fd29-4177-93b1-f26c23409fec-kube-api-access-tchtz\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.212740 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.212707    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-dshm\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.212989 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.212778    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-model-cache\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.212989 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.212871    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-home\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.213182 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.213137    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-tmp-dir\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.213434 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.213271    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d268d7a-fd29-4177-93b1-f26c23409fec-tls-certs\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.315107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.315059    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-model-cache\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.315316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.315129    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-home\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.315426 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.315409    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-tmp-dir\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.315516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.315501    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d268d7a-fd29-4177-93b1-f26c23409fec-tls-certs\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.315579 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.315531    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-home\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.315633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.315592    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-model-cache\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.315633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.315602    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tchtz\" (UniqueName: \"kubernetes.io/projected/4d268d7a-fd29-4177-93b1-f26c23409fec-kube-api-access-tchtz\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.315736 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.315671    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-dshm\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.316159 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.316133    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-tmp-dir\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.318640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.318614    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-dshm\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.318835 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.318819    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d268d7a-fd29-4177-93b1-f26c23409fec-tls-certs\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.330192 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.330164    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tchtz\" (UniqueName: \"kubernetes.io/projected/4d268d7a-fd29-4177-93b1-f26c23409fec-kube-api-access-tchtz\") pod \"e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") " pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.384332 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.383760    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"]
Nov 02 09:52:00.390568 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.390537    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"
Nov 02 09:52:00.409521 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.409379    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"]
Nov 02 09:52:00.412123 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.412090    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:52:00.418266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.415080    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:52:00.418266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.415106    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:52:00.418266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.415455    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:52:00.476996 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.476935    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:00.521391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.521351    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63067f92-nrvtz\" (UID: \"a29a1545-bff1-42a5-9332-df3922620bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"
Nov 02 09:52:00.521548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.521478    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cspkh\" (UniqueName: \"kubernetes.io/projected/a29a1545-bff1-42a5-9332-df3922620bb1-kube-api-access-cspkh\") pod \"maas-api-revoke-sub-63067f92-nrvtz\" (UID: \"a29a1545-bff1-42a5-9332-df3922620bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"
Nov 02 09:52:00.521548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.521538    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63067f92-nrvtz\" (UID: \"a29a1545-bff1-42a5-9332-df3922620bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"
Nov 02 09:52:00.623297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.623250    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63067f92-nrvtz\" (UID: \"a29a1545-bff1-42a5-9332-df3922620bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"
Nov 02 09:52:00.623479 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.623368    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cspkh\" (UniqueName: \"kubernetes.io/projected/a29a1545-bff1-42a5-9332-df3922620bb1-kube-api-access-cspkh\") pod \"maas-api-revoke-sub-63067f92-nrvtz\" (UID: \"a29a1545-bff1-42a5-9332-df3922620bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"
Nov 02 09:52:00.623479 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.623449    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63067f92-nrvtz\" (UID: \"a29a1545-bff1-42a5-9332-df3922620bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"
Nov 02 09:52:00.626591 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.626544    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63067f92-nrvtz\" (UID: \"a29a1545-bff1-42a5-9332-df3922620bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"
Nov 02 09:52:00.627560 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.627520    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63067f92-nrvtz\" (UID: \"a29a1545-bff1-42a5-9332-df3922620bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"
Nov 02 09:52:00.638241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.638176    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cspkh\" (UniqueName: \"kubernetes.io/projected/a29a1545-bff1-42a5-9332-df3922620bb1-kube-api-access-cspkh\") pod \"maas-api-revoke-sub-63067f92-nrvtz\" (UID: \"a29a1545-bff1-42a5-9332-df3922620bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"
Nov 02 09:52:00.746515 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.746426    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"
Nov 02 09:52:00.806257 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.806200    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"]
Nov 02 09:52:00.823771 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:52:00.816447    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d268d7a_fd29_4177_93b1_f26c23409fec.slice/crio-795b323967c21d0ad0710e04f8be98f4f77ba1fb0925dc4a960c6977c2da06b0 WatchSource:0}: Error finding container 795b323967c21d0ad0710e04f8be98f4f77ba1fb0925dc4a960c6977c2da06b0: Status 404 returned error can't find the container with id 795b323967c21d0ad0710e04f8be98f4f77ba1fb0925dc4a960c6977c2da06b0
Nov 02 09:52:00.971706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:00.971578    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs" event={"ID":"4d268d7a-fd29-4177-93b1-f26c23409fec","Type":"ContainerStarted","Data":"795b323967c21d0ad0710e04f8be98f4f77ba1fb0925dc4a960c6977c2da06b0"}
Nov 02 09:52:01.075600 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:01.075568    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"]
Nov 02 09:52:02.066880 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.066836    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs" event={"ID":"4d268d7a-fd29-4177-93b1-f26c23409fec","Type":"ContainerStarted","Data":"445e0209e77338f8a68a2e197536f0770792f4937bd3de4c492a51d298f1fc14"}
Nov 02 09:52:02.085478 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.085437    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz" event={"ID":"a29a1545-bff1-42a5-9332-df3922620bb1","Type":"ContainerStarted","Data":"ea79ac4fbe4badf97290017a5faf64684995d4546838f450ab50fa66fac7d586"}
Nov 02 09:52:02.085731 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.085707    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz" event={"ID":"a29a1545-bff1-42a5-9332-df3922620bb1","Type":"ContainerStarted","Data":"c867a772d56a52dbbcf0d8e53d8dd32808350e1d7354e5a454b19d8bf008451e"}
Nov 02 09:52:02.140345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.135230    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs" podStartSLOduration=2.1351927059999998 podStartE2EDuration="2.135192706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:02.106824128 +0000 UTC m=+999.045110904" watchObservedRunningTime="2026-11-02 09:52:02.135192706 +0000 UTC m=+999.073479483"
Nov 02 09:52:02.145241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.142503    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz" podStartSLOduration=2.142483486 podStartE2EDuration="2.142483486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:02.127617969 +0000 UTC m=+999.065904767" watchObservedRunningTime="2026-11-02 09:52:02.142483486 +0000 UTC m=+999.080770262"
Nov 02 09:52:02.199040 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.198393    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main" containerID="cri-o://9549dcf35923b7245a92febf590db5190b9c6513ac282fbc1414b94f20e55793" gracePeriod=45
Nov 02 09:52:02.309316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.309275    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"]
Nov 02 09:52:02.421655 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.421619    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"]
Nov 02 09:52:02.425752 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.423349    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"
Nov 02 09:52:02.488399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.486462    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:02.535733 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.535697    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cba9d18-29a5-4001-8970-eef712944669-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4c342351-wztjj\" (UID: \"0cba9d18-29a5-4001-8970-eef712944669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"
Nov 02 09:52:02.536352 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.536325    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjgj2\" (UniqueName: \"kubernetes.io/projected/0cba9d18-29a5-4001-8970-eef712944669-kube-api-access-gjgj2\") pod \"maas-api-revoke-keys-e2e-migrate-4c342351-wztjj\" (UID: \"0cba9d18-29a5-4001-8970-eef712944669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"
Nov 02 09:52:02.536707 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.536687    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cba9d18-29a5-4001-8970-eef712944669-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4c342351-wztjj\" (UID: \"0cba9d18-29a5-4001-8970-eef712944669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"
Nov 02 09:52:02.638466 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.638419    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cba9d18-29a5-4001-8970-eef712944669-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4c342351-wztjj\" (UID: \"0cba9d18-29a5-4001-8970-eef712944669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"
Nov 02 09:52:02.638637 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.638525    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjgj2\" (UniqueName: \"kubernetes.io/projected/0cba9d18-29a5-4001-8970-eef712944669-kube-api-access-gjgj2\") pod \"maas-api-revoke-keys-e2e-migrate-4c342351-wztjj\" (UID: \"0cba9d18-29a5-4001-8970-eef712944669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"
Nov 02 09:52:02.638711 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.638666    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cba9d18-29a5-4001-8970-eef712944669-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4c342351-wztjj\" (UID: \"0cba9d18-29a5-4001-8970-eef712944669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"
Nov 02 09:52:02.639539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.639512    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cba9d18-29a5-4001-8970-eef712944669-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4c342351-wztjj\" (UID: \"0cba9d18-29a5-4001-8970-eef712944669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"
Nov 02 09:52:02.641554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.641497    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cba9d18-29a5-4001-8970-eef712944669-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4c342351-wztjj\" (UID: \"0cba9d18-29a5-4001-8970-eef712944669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"
Nov 02 09:52:02.683993 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.683919    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjgj2\" (UniqueName: \"kubernetes.io/projected/0cba9d18-29a5-4001-8970-eef712944669-kube-api-access-gjgj2\") pod \"maas-api-revoke-keys-e2e-migrate-4c342351-wztjj\" (UID: \"0cba9d18-29a5-4001-8970-eef712944669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"
Nov 02 09:52:02.712133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.712104    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:52:02.739823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.739793    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-dshm\") pod \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") "
Nov 02 09:52:02.744449 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.744406    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-kube-api-access-2qcnf\" (UniqueName: \"kubernetes.io/projected/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-kube-api-access-2qcnf\") pod \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") "
Nov 02 09:52:02.744587 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.744483    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tls-certs\") pod \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") "
Nov 02 09:52:02.744587 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.744508    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-dshm" pod "f6a56df7-3e4e-421c-808b-c2fd2f440ea4" (UID: "f6a56df7-3e4e-421c-808b-c2fd2f440ea4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:02.744587 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.744540    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tmp-dir\") pod \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") "
Nov 02 09:52:02.744769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.744611    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-model-cache\") pod \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") "
Nov 02 09:52:02.744769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.744692    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-home\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-home\") pod \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\" (UID: \"f6a56df7-3e4e-421c-808b-c2fd2f440ea4\") "
Nov 02 09:52:02.745083 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.745034    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-model-cache" pod "f6a56df7-3e4e-421c-808b-c2fd2f440ea4" (UID: "f6a56df7-3e4e-421c-808b-c2fd2f440ea4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:02.745299 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.745267    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-home" pod "f6a56df7-3e4e-421c-808b-c2fd2f440ea4" (UID: "f6a56df7-3e4e-421c-808b-c2fd2f440ea4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:02.745620 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.745603    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tmp-dir" pod "f6a56df7-3e4e-421c-808b-c2fd2f440ea4" (UID: "f6a56df7-3e4e-421c-808b-c2fd2f440ea4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:02.745742 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.745715    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:02.745809 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.745752    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:02.745809 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.745766    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:02.792093 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.792060    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"
Nov 02 09:52:02.797702 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.797656    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-kube-api-access-2qcnf" pod "f6a56df7-3e4e-421c-808b-c2fd2f440ea4" (UID: "f6a56df7-3e4e-421c-808b-c2fd2f440ea4"). InnerVolumeSpecName "kube-api-access-2qcnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:02.805285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.805252    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tls-certs" pod "f6a56df7-3e4e-421c-808b-c2fd2f440ea4" (UID: "f6a56df7-3e4e-421c-808b-c2fd2f440ea4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:52:02.848800 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.848722    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qcnf\" (UniqueName: \"kubernetes.io/projected/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-kube-api-access-2qcnf\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:02.848800 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.848758    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:02.848800 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:02.848774    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a56df7-3e4e-421c-808b-c2fd2f440ea4-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:03.083831 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.083704    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main" containerID="cri-o://4034954677b3d4bbc4a02e1a6648868af634025219ee0c74c1060c1a1cecf137" gracePeriod=45
Nov 02 09:52:03.104247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.102138    4449 generic.go:356] "Generic (PLEG): container finished" podID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerID="9549dcf35923b7245a92febf590db5190b9c6513ac282fbc1414b94f20e55793" exitCode=0
Nov 02 09:52:03.104247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.102546    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" event={"ID":"f6a56df7-3e4e-421c-808b-c2fd2f440ea4","Type":"ContainerDied","Data":"9549dcf35923b7245a92febf590db5190b9c6513ac282fbc1414b94f20e55793"}
Nov 02 09:52:03.104247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.102591    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz" event={"ID":"f6a56df7-3e4e-421c-808b-c2fd2f440ea4","Type":"ContainerDied","Data":"42a4acae36ec9ada7a85a4ee682503c4d9b84da1086352520d65b1c68a657450"}
Nov 02 09:52:03.104247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.102616    4449 scope.go:122] "RemoveContainer" containerID="9549dcf35923b7245a92febf590db5190b9c6513ac282fbc1414b94f20e55793"
Nov 02 09:52:03.104247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.102822    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"
Nov 02 09:52:03.107056 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.105398    4449 generic.go:356] "Generic (PLEG): container finished" podID="a29a1545-bff1-42a5-9332-df3922620bb1" containerID="ea79ac4fbe4badf97290017a5faf64684995d4546838f450ab50fa66fac7d586" exitCode=0
Nov 02 09:52:03.107056 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.106188    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz" event={"ID":"a29a1545-bff1-42a5-9332-df3922620bb1","Type":"ContainerDied","Data":"ea79ac4fbe4badf97290017a5faf64684995d4546838f450ab50fa66fac7d586"}
Nov 02 09:52:03.149783 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.149645    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"]
Nov 02 09:52:03.181492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.181401    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"]
Nov 02 09:52:03.184887 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:52:03.184855    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cba9d18_29a5_4001_8970_eef712944669.slice/crio-eb71e7fb755d7c1ee8ed1727d40a983f45eb9df94a3c0fd81c07ee17749a1382 WatchSource:0}: Error finding container eb71e7fb755d7c1ee8ed1727d40a983f45eb9df94a3c0fd81c07ee17749a1382: Status 404 returned error can't find the container with id eb71e7fb755d7c1ee8ed1727d40a983f45eb9df94a3c0fd81c07ee17749a1382
Nov 02 09:52:03.187705 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.187679    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/rate-test-model-bde944-kserve-5df8884f97-6xflz"]
Nov 02 09:52:03.203802 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.203718    4449 scope.go:122] "RemoveContainer" containerID="9549dcf35923b7245a92febf590db5190b9c6513ac282fbc1414b94f20e55793"
Nov 02 09:52:03.204241 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:52:03.204192    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9549dcf35923b7245a92febf590db5190b9c6513ac282fbc1414b94f20e55793\": container with ID starting with 9549dcf35923b7245a92febf590db5190b9c6513ac282fbc1414b94f20e55793 not found: ID does not exist" containerID="9549dcf35923b7245a92febf590db5190b9c6513ac282fbc1414b94f20e55793"
Nov 02 09:52:03.205332 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.205260    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9549dcf35923b7245a92febf590db5190b9c6513ac282fbc1414b94f20e55793"} err="failed to get container status \"9549dcf35923b7245a92febf590db5190b9c6513ac282fbc1414b94f20e55793\": rpc error: code = NotFound desc = could not find container \"9549dcf35923b7245a92febf590db5190b9c6513ac282fbc1414b94f20e55793\": container with ID starting with 9549dcf35923b7245a92febf590db5190b9c6513ac282fbc1414b94f20e55793 not found: ID does not exist"
Nov 02 09:52:03.217232 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.217185    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" path="/var/lib/kubelet/pods/f6a56df7-3e4e-421c-808b-c2fd2f440ea4/volumes"
Nov 02 09:52:03.397888 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.397819    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"]
Nov 02 09:52:03.399331 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.399246    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main"
Nov 02 09:52:03.405857 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.400789    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main"
Nov 02 09:52:03.405857 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.404499    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a56df7-3e4e-421c-808b-c2fd2f440ea4" containerName="main"
Nov 02 09:52:03.411188 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.410701    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"]
Nov 02 09:52:03.411188 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.410816    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.414415 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.414336    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.492481    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tls-certs\") pod \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") "
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.492600    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-home\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-home\") pod \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") "
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.492634    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-model-cache\") pod \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") "
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.492659    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-dshm\") pod \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") "
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.492688    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-kube-api-access-6w24d\" (UniqueName: \"kubernetes.io/projected/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-kube-api-access-6w24d\") pod \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") "
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.492738    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tmp-dir\") pod \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\" (UID: \"72db0ef3-c161-4bf2-b844-a8aabb71c8bc\") "
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.492897    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.492915    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.492934    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.492979    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6h4\" (UniqueName: \"kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-kube-api-access-xl6h4\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.493053    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.493113    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ac13cf-aa96-4117-825a-8b4aba00c77e-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.493134    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.493161    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.493344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.493271    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.494243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.493922    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-home" pod "72db0ef3-c161-4bf2-b844-a8aabb71c8bc" (UID: "72db0ef3-c161-4bf2-b844-a8aabb71c8bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:03.494243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.494098    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-model-cache" pod "72db0ef3-c161-4bf2-b844-a8aabb71c8bc" (UID: "72db0ef3-c161-4bf2-b844-a8aabb71c8bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:03.494731 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.494693    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tmp-dir" pod "72db0ef3-c161-4bf2-b844-a8aabb71c8bc" (UID: "72db0ef3-c161-4bf2-b844-a8aabb71c8bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:03.498488 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.498174    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-dshm" pod "72db0ef3-c161-4bf2-b844-a8aabb71c8bc" (UID: "72db0ef3-c161-4bf2-b844-a8aabb71c8bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:03.498488 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.498298    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-kube-api-access-6w24d" pod "72db0ef3-c161-4bf2-b844-a8aabb71c8bc" (UID: "72db0ef3-c161-4bf2-b844-a8aabb71c8bc"). InnerVolumeSpecName "kube-api-access-6w24d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:03.507810 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.507774    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tls-certs" pod "72db0ef3-c161-4bf2-b844-a8aabb71c8bc" (UID: "72db0ef3-c161-4bf2-b844-a8aabb71c8bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:52:03.594382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.594331    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.594569 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.594437    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.594569 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.594465    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.594569 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.594495    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.594747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.594570    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6h4\" (UniqueName: \"kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-kube-api-access-xl6h4\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.594747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.594683    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.594864 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.594775    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ac13cf-aa96-4117-825a-8b4aba00c77e-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.594864 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.594805    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.594864 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.594838    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.595092 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.594844    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.595092 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.594948    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.595195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.595112    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.595363 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.595339    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:03.595531 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.595373    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:03.595531 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.595389    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:03.595531 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.595407    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:03.595531 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.595419    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:03.595531 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.595446    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w24d\" (UniqueName: \"kubernetes.io/projected/72db0ef3-c161-4bf2-b844-a8aabb71c8bc-kube-api-access-6w24d\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:03.595719 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.595561    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.596047 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.596024    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ac13cf-aa96-4117-825a-8b4aba00c77e-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.597954 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.597931    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.598199 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.598181    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.607912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.607884    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6h4\" (UniqueName: \"kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-kube-api-access-xl6h4\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.608007 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.607938    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:03.738716 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:03.738678    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:04.117281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.117249    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"]
Nov 02 09:52:04.122196 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.122153    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:52:04.122357 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.122276    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:52:04.122357 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.122340    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:52:04.122857 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.122812    4449 generic.go:356] "Generic (PLEG): container finished" podID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerID="4034954677b3d4bbc4a02e1a6648868af634025219ee0c74c1060c1a1cecf137" exitCode=0
Nov 02 09:52:04.122990 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.122920    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" event={"ID":"72db0ef3-c161-4bf2-b844-a8aabb71c8bc","Type":"ContainerDied","Data":"4034954677b3d4bbc4a02e1a6648868af634025219ee0c74c1060c1a1cecf137"}
Nov 02 09:52:04.122990 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.122954    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9" event={"ID":"72db0ef3-c161-4bf2-b844-a8aabb71c8bc","Type":"ContainerDied","Data":"30b0410af32094804abf45425c1c5ac2ede6cca95151ccad15ffd68557180851"}
Nov 02 09:52:04.122990 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.122975    4449 scope.go:122] "RemoveContainer" containerID="4034954677b3d4bbc4a02e1a6648868af634025219ee0c74c1060c1a1cecf137"
Nov 02 09:52:04.122990 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.122972    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"
Nov 02 09:52:04.134552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.134129    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj" event={"ID":"0cba9d18-29a5-4001-8970-eef712944669","Type":"ContainerStarted","Data":"5d421fdfd4aefdd55732f9e2a5a28fa1d808731e1c5937419d58225da8744015"}
Nov 02 09:52:04.134552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.134182    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj" event={"ID":"0cba9d18-29a5-4001-8970-eef712944669","Type":"ContainerStarted","Data":"eb71e7fb755d7c1ee8ed1727d40a983f45eb9df94a3c0fd81c07ee17749a1382"}
Nov 02 09:52:04.145912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.145873    4449 scope.go:122] "RemoveContainer" containerID="4034954677b3d4bbc4a02e1a6648868af634025219ee0c74c1060c1a1cecf137"
Nov 02 09:52:04.146369 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:52:04.146321    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4034954677b3d4bbc4a02e1a6648868af634025219ee0c74c1060c1a1cecf137\": container with ID starting with 4034954677b3d4bbc4a02e1a6648868af634025219ee0c74c1060c1a1cecf137 not found: ID does not exist" containerID="4034954677b3d4bbc4a02e1a6648868af634025219ee0c74c1060c1a1cecf137"
Nov 02 09:52:04.146369 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.146350    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4034954677b3d4bbc4a02e1a6648868af634025219ee0c74c1060c1a1cecf137"} err="failed to get container status \"4034954677b3d4bbc4a02e1a6648868af634025219ee0c74c1060c1a1cecf137\": rpc error: code = NotFound desc = could not find container \"4034954677b3d4bbc4a02e1a6648868af634025219ee0c74c1060c1a1cecf137\": container with ID starting with 4034954677b3d4bbc4a02e1a6648868af634025219ee0c74c1060c1a1cecf137 not found: ID does not exist"
Nov 02 09:52:04.187558 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.186732    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj" podStartSLOduration=2.186718173 podStartE2EDuration="2.186718173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:04.180594133 +0000 UTC m=+1001.118880936" watchObservedRunningTime="2026-11-02 09:52:04.186718173 +0000 UTC m=+1001.125005005"
Nov 02 09:52:04.207177 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.206559    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"]
Nov 02 09:52:04.215430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.215402    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/rate-test-model-9e7156-kserve-5d46dd64c7-9g2k9"]
Nov 02 09:52:04.395034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.395006    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"
Nov 02 09:52:04.508971 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.508644    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-xks-ca\") pod \"a29a1545-bff1-42a5-9332-df3922620bb1\" (UID: \"a29a1545-bff1-42a5-9332-df3922620bb1\") "
Nov 02 09:52:04.508971 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.508754    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29a1545-bff1-42a5-9332-df3922620bb1-kube-api-access-cspkh\" (UniqueName: \"kubernetes.io/projected/a29a1545-bff1-42a5-9332-df3922620bb1-kube-api-access-cspkh\") pod \"a29a1545-bff1-42a5-9332-df3922620bb1\" (UID: \"a29a1545-bff1-42a5-9332-df3922620bb1\") "
Nov 02 09:52:04.509194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.508980    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-openshift-ca\") pod \"a29a1545-bff1-42a5-9332-df3922620bb1\" (UID: \"a29a1545-bff1-42a5-9332-df3922620bb1\") "
Nov 02 09:52:04.509759 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.509370    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-openshift-ca" pod "a29a1545-bff1-42a5-9332-df3922620bb1" (UID: "a29a1545-bff1-42a5-9332-df3922620bb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:52:04.509759 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.509584    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:04.512438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.512400    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29a1545-bff1-42a5-9332-df3922620bb1-kube-api-access-cspkh" pod "a29a1545-bff1-42a5-9332-df3922620bb1" (UID: "a29a1545-bff1-42a5-9332-df3922620bb1"). InnerVolumeSpecName "kube-api-access-cspkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:04.515260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.515235    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-xks-ca" pod "a29a1545-bff1-42a5-9332-df3922620bb1" (UID: "a29a1545-bff1-42a5-9332-df3922620bb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:52:04.611411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.611368    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29a1545-bff1-42a5-9332-df3922620bb1-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:04.611411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:04.611408    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cspkh\" (UniqueName: \"kubernetes.io/projected/a29a1545-bff1-42a5-9332-df3922620bb1-kube-api-access-cspkh\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:05.035904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.035853    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:52:05.049131 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.049090    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:52:05.206238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.198334    4449 generic.go:356] "Generic (PLEG): container finished" podID="0cba9d18-29a5-4001-8970-eef712944669" containerID="5d421fdfd4aefdd55732f9e2a5a28fa1d808731e1c5937419d58225da8744015" exitCode=0
Nov 02 09:52:05.206238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.199030    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj" event={"ID":"0cba9d18-29a5-4001-8970-eef712944669","Type":"ContainerDied","Data":"5d421fdfd4aefdd55732f9e2a5a28fa1d808731e1c5937419d58225da8744015"}
Nov 02 09:52:05.207547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.207309    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"
Nov 02 09:52:05.212762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.212738    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" path="/var/lib/kubelet/pods/72db0ef3-c161-4bf2-b844-a8aabb71c8bc/volumes"
Nov 02 09:52:05.213273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.213255    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz" event={"ID":"a29a1545-bff1-42a5-9332-df3922620bb1","Type":"ContainerDied","Data":"c867a772d56a52dbbcf0d8e53d8dd32808350e1d7354e5a454b19d8bf008451e"}
Nov 02 09:52:05.213320 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.213294    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c867a772d56a52dbbcf0d8e53d8dd32808350e1d7354e5a454b19d8bf008451e"
Nov 02 09:52:05.213401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.213388    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4" event={"ID":"58ac13cf-aa96-4117-825a-8b4aba00c77e","Type":"ContainerStarted","Data":"6306f837eb49c430bdcb2027748be76b27bc7e57788711067a3d32ff72ce6b75"}
Nov 02 09:52:05.213446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.213416    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4" event={"ID":"58ac13cf-aa96-4117-825a-8b4aba00c77e","Type":"ContainerStarted","Data":"f82f15d799df41713eb0c3cea2a1d59f864ab766d8774ac67710942b104f4caa"}
Nov 02 09:52:05.228315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.228295    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:52:05.247307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.247257    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4" podStartSLOduration=2.247240054 podStartE2EDuration="2.247240054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:05.241921146 +0000 UTC m=+1002.180207925" watchObservedRunningTime="2026-11-02 09:52:05.247240054 +0000 UTC m=+1002.185526827"
Nov 02 09:52:05.739554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.739523    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:05.741363 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.741328    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 09:52:05.741533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:05.741377    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4" podUID="58ac13cf-aa96-4117-825a-8b4aba00c77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 09:52:06.612285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:06.610235    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"
Nov 02 09:52:06.634123 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:52:06.634083    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:52:06.710120 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:06.710090    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cba9d18-29a5-4001-8970-eef712944669-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cba9d18-29a5-4001-8970-eef712944669-maas-api-openshift-ca\") pod \"0cba9d18-29a5-4001-8970-eef712944669\" (UID: \"0cba9d18-29a5-4001-8970-eef712944669\") "
Nov 02 09:52:06.710308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:06.710146    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cba9d18-29a5-4001-8970-eef712944669-kube-api-access-gjgj2\" (UniqueName: \"kubernetes.io/projected/0cba9d18-29a5-4001-8970-eef712944669-kube-api-access-gjgj2\") pod \"0cba9d18-29a5-4001-8970-eef712944669\" (UID: \"0cba9d18-29a5-4001-8970-eef712944669\") "
Nov 02 09:52:06.710356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:06.710307    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cba9d18-29a5-4001-8970-eef712944669-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cba9d18-29a5-4001-8970-eef712944669-maas-api-xks-ca\") pod \"0cba9d18-29a5-4001-8970-eef712944669\" (UID: \"0cba9d18-29a5-4001-8970-eef712944669\") "
Nov 02 09:52:06.711614 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:06.711574    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cba9d18-29a5-4001-8970-eef712944669-maas-api-openshift-ca" pod "0cba9d18-29a5-4001-8970-eef712944669" (UID: "0cba9d18-29a5-4001-8970-eef712944669"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:52:06.713434 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:06.713392    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cba9d18-29a5-4001-8970-eef712944669-maas-api-xks-ca" pod "0cba9d18-29a5-4001-8970-eef712944669" (UID: "0cba9d18-29a5-4001-8970-eef712944669"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:52:06.714145 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:06.714116    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cba9d18-29a5-4001-8970-eef712944669-kube-api-access-gjgj2" pod "0cba9d18-29a5-4001-8970-eef712944669" (UID: "0cba9d18-29a5-4001-8970-eef712944669"). InnerVolumeSpecName "kube-api-access-gjgj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:06.739826 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:06.739794    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 09:52:06.739959 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:06.739845    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4" podUID="58ac13cf-aa96-4117-825a-8b4aba00c77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 09:52:06.813539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:06.813508    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cba9d18-29a5-4001-8970-eef712944669-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:06.813539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:06.813535    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjgj2\" (UniqueName: \"kubernetes.io/projected/0cba9d18-29a5-4001-8970-eef712944669-kube-api-access-gjgj2\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:06.813539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:06.813545    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cba9d18-29a5-4001-8970-eef712944669-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:07.247012 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.246984    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"
Nov 02 09:52:07.247274 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.247057    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj" event={"ID":"0cba9d18-29a5-4001-8970-eef712944669","Type":"ContainerDied","Data":"eb71e7fb755d7c1ee8ed1727d40a983f45eb9df94a3c0fd81c07ee17749a1382"}
Nov 02 09:52:07.247274 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.247111    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb71e7fb755d7c1ee8ed1727d40a983f45eb9df94a3c0fd81c07ee17749a1382"
Nov 02 09:52:07.273894 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.271646    4449 generic.go:356] "Generic (PLEG): container finished" podID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerID="85827e0be84c335ba8f2d9ee3460f4aa50669a03f14e148a739074408c35331b" exitCode=0
Nov 02 09:52:07.273894 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.272391    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" event={"ID":"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc","Type":"ContainerDied","Data":"85827e0be84c335ba8f2d9ee3460f4aa50669a03f14e148a739074408c35331b"}
Nov 02 09:52:07.273894 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.273130    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:52:07.315641 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.302420    4449 scope.go:122] "RemoveContainer" containerID="85827e0be84c335ba8f2d9ee3460f4aa50669a03f14e148a739074408c35331b"
Nov 02 09:52:07.322301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.317190    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:52:07.322301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.317300    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:52:07.322301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.317341    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:52:07.739699 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.739659    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 09:52:07.740437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.739717    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4" podUID="58ac13cf-aa96-4117-825a-8b4aba00c77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 09:52:07.791187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.791152    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"]
Nov 02 09:52:07.792067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.792046    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0cba9d18-29a5-4001-8970-eef712944669" containerName="revoke-keys"
Nov 02 09:52:07.792119 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.792071    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cba9d18-29a5-4001-8970-eef712944669" containerName="revoke-keys"
Nov 02 09:52:07.792119 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.792098    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a29a1545-bff1-42a5-9332-df3922620bb1" containerName="revoke-keys"
Nov 02 09:52:07.792119 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.792103    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29a1545-bff1-42a5-9332-df3922620bb1" containerName="revoke-keys"
Nov 02 09:52:07.792119 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.792116    4449 cpu_manager.go:414] "containerMap: removing container" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main"
Nov 02 09:52:07.792276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.792121    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main"
Nov 02 09:52:07.792276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.792253    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cba9d18-29a5-4001-8970-eef712944669" containerName="revoke-keys"
Nov 02 09:52:07.792276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.792265    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29a1545-bff1-42a5-9332-df3922620bb1" containerName="revoke-keys"
Nov 02 09:52:07.792276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.792271    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="72db0ef3-c161-4bf2-b844-a8aabb71c8bc" containerName="main"
Nov 02 09:52:07.797152 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.797039    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.813705 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.813612    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"]
Nov 02 09:52:07.838517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.837732    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.838517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.837863    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.838517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.837908    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.838517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.837941    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.838517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.837971    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.838517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.837999    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.838517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.838064    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn84l\" (UniqueName: \"kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-kube-api-access-dn84l\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.838517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.838129    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.838517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.838209    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.940049 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.940014    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.940257 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.940102    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.940257 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.940198    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.940396 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.940314    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.940396 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.940361    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.940514 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.940392    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.940514 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.940432    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.940514 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.940501    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.940666 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.940513    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.940666 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.940570    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.940666 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.940596    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn84l\" (UniqueName: \"kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-kube-api-access-dn84l\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.940818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.940711    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.941056 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.941024    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.941145 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.941070    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.943066 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.943039    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.943334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.943316    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.984121 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.984082    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn84l\" (UniqueName: \"kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-kube-api-access-dn84l\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:07.984293 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:07.984084    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:08.037261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:08.035401    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"]
Nov 02 09:52:08.043021 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:08.042936    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0054f1-bg5ng"]
Nov 02 09:52:08.132294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:08.132075    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:08.323838 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:08.323802    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" event={"ID":"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc","Type":"ContainerStarted","Data":"8cc48100801023880a7d78c42a0f546872a153e2b62a66eeef3adb335aa1e692"}
Nov 02 09:52:08.518496 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:52:08.518453    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbc7509_08cd_45ab_ad9d_32527a7c48af.slice/crio-95cb4e8e88e85a72d6752a010374c8de1b361bb8d1c30a2f98efb24c11bdfb0f WatchSource:0}: Error finding container 95cb4e8e88e85a72d6752a010374c8de1b361bb8d1c30a2f98efb24c11bdfb0f: Status 404 returned error can't find the container with id 95cb4e8e88e85a72d6752a010374c8de1b361bb8d1c30a2f98efb24c11bdfb0f
Nov 02 09:52:08.521253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:08.521197    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:52:08.521345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:08.521290    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:52:08.521345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:08.521320    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:52:08.522092 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:08.521949    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"]
Nov 02 09:52:08.739270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:08.739204    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 09:52:08.739467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:08.739294    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4" podUID="58ac13cf-aa96-4117-825a-8b4aba00c77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 09:52:09.217997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:09.217966    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0452787b-5cbb-492a-9c7c-7e36a6b6e41c" path="/var/lib/kubelet/pods/0452787b-5cbb-492a-9c7c-7e36a6b6e41c/volumes"
Nov 02 09:52:09.314711 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:09.314675    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:52:09.316419 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:09.316392    4449 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 02 09:52:09.316557 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:09.316471    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 02 09:52:09.335167 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:09.335137    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc" event={"ID":"bdbc7509-08cd-45ab-ad9d-32527a7c48af","Type":"ContainerStarted","Data":"547acb298827aa6664e7cf78af29280b222bc99176569af6ea09ff61e8f178f6"}
Nov 02 09:52:09.335335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:09.335198    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc" event={"ID":"bdbc7509-08cd-45ab-ad9d-32527a7c48af","Type":"ContainerStarted","Data":"95cb4e8e88e85a72d6752a010374c8de1b361bb8d1c30a2f98efb24c11bdfb0f"}
Nov 02 09:52:09.393244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:09.385602    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc" podStartSLOduration=2.385577489 podStartE2EDuration="2.385577489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:09.367622283 +0000 UTC m=+1006.305909060" watchObservedRunningTime="2026-11-02 09:52:09.385577489 +0000 UTC m=+1006.323864266"
Nov 02 09:52:09.743092 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:09.742645    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 09:52:09.743092 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:09.742707    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4" podUID="58ac13cf-aa96-4117-825a-8b4aba00c77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 09:52:10.135347 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:10.135307    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:10.136415 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:10.136386    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 02 09:52:10.136564 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:10.136459    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc" podUID="bdbc7509-08cd-45ab-ad9d-32527a7c48af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 02 09:52:10.327243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:10.320717    4449 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 02 09:52:10.327243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:10.320780    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 02 09:52:10.495299 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:10.483297    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:10.639241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:10.639051    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:10.739844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:10.739805    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 09:52:10.740020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:10.739859    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4" podUID="58ac13cf-aa96-4117-825a-8b4aba00c77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 09:52:11.047841 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.047797    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"]
Nov 02 09:52:11.126692 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.126659    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.139369 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.139332    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 02 09:52:11.139686 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.139651    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc" podUID="bdbc7509-08cd-45ab-ad9d-32527a7c48af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 02 09:52:11.153244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.147889    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9e7156\"/\"openshift-service-ca.crt\""
Nov 02 09:52:11.153244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.148707    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9e7156\"/\"default-dockercfg-8h4tz\""
Nov 02 09:52:11.153244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.149055    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9e7156\"/\"sub-test-model-9e7156-kserve-self-signed-certs\""
Nov 02 09:52:11.153244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.149082    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9e7156\"/\"kube-root-ca.crt\""
Nov 02 09:52:11.225909 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.223432    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"]
Nov 02 09:52:11.303298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.303156    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gh7v\" (UniqueName: \"kubernetes.io/projected/7968e57e-6e72-4ef2-af9f-82c111d870f8-kube-api-access-6gh7v\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.303298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.303263    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-tmp-dir\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.303543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.303332    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-home\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.303543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.303397    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-model-cache\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.303543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.303464    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-dshm\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.303696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.303630    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7968e57e-6e72-4ef2-af9f-82c111d870f8-tls-certs\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.315625 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.315164    4449 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 02 09:52:11.315625 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.315254    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 02 09:52:11.434572 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.434536    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7968e57e-6e72-4ef2-af9f-82c111d870f8-tls-certs\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.439242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.435284    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gh7v\" (UniqueName: \"kubernetes.io/projected/7968e57e-6e72-4ef2-af9f-82c111d870f8-kube-api-access-6gh7v\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.447244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.439566    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-tmp-dir\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.469242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.447624    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-home\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.469640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.469608    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-model-cache\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.469829 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.469815    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-dshm\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.470046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.450895    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-home\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.470146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.442922    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-tmp-dir\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.470263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.470044    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-model-cache\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.470360 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.442535    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7968e57e-6e72-4ef2-af9f-82c111d870f8-tls-certs\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.478238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.472098    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gh7v\" (UniqueName: \"kubernetes.io/projected/7968e57e-6e72-4ef2-af9f-82c111d870f8-kube-api-access-6gh7v\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.487604 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.487530    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-dshm\") pod \"sub-test-model-9e7156-kserve-74bb6878f5-rn88n\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.517980 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.517672    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:11.546306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:11.546276    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:52:12.091243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:12.078568    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:12.091243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:12.080310    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:12.136279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:12.122801    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 09:52:12.145106 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:12.144449    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 02 09:52:12.145106 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:12.144533    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc" podUID="bdbc7509-08cd-45ab-ad9d-32527a7c48af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 02 09:52:12.330315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:12.325865    4449 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 02 09:52:12.330315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:12.325945    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 02 09:52:12.381256 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:12.374182    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"]
Nov 02 09:52:12.433255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:12.432994    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n" event={"ID":"7968e57e-6e72-4ef2-af9f-82c111d870f8","Type":"ContainerStarted","Data":"6e96678041dde005fc27e2e8f01891aae8916db076dcd4b251c5b3388549befb"}
Nov 02 09:52:12.441394 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:12.441352    4449 patch_prober.go:29] interesting pod/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:52:12.441867 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:12.441416    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj" podUID="1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:52:12.620355 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:12.614575    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"]
Nov 02 09:52:13.133865 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:13.133431    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 02 09:52:13.133865 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:13.133494    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc" podUID="bdbc7509-08cd-45ab-ad9d-32527a7c48af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 02 09:52:13.314529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:13.314495    4449 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 02 09:52:13.314766 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:13.314745    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 02 09:52:13.447793 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:13.447699    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n" event={"ID":"7968e57e-6e72-4ef2-af9f-82c111d870f8","Type":"ContainerStarted","Data":"7526d32fbc216ad1d9888229b3a6895310c12504ef4b97814935fdd8b8125c77"}
Nov 02 09:52:13.496245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:13.496047    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n" podStartSLOduration=2.496027894 podStartE2EDuration="2.496027894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:13.481180588 +0000 UTC m=+1010.419467363" watchObservedRunningTime="2026-11-02 09:52:13.496027894 +0000 UTC m=+1010.434314670"
Nov 02 09:52:13.523861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:13.522327    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:14.133245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:14.133060    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 02 09:52:14.133245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:14.133125    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc" podUID="bdbc7509-08cd-45ab-ad9d-32527a7c48af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 02 09:52:14.916264 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:14.899273    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"]
Nov 02 09:52:14.997245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:14.974541    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"
Nov 02 09:52:15.306303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.068539    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:52:15.306303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.068663    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:52:15.306303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.068710    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:52:15.306303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.068758    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:52:15.306303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.199624    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v25p\" (UniqueName: \"kubernetes.io/projected/13359001-d62c-44cb-bcbc-400474ff34de-kube-api-access-4v25p\") pod \"maas-api-revoke-sub-a3f77675-gnslw\" (UID: \"13359001-d62c-44cb-bcbc-400474ff34de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"
Nov 02 09:52:15.306303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.199845    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f77675-gnslw\" (UID: \"13359001-d62c-44cb-bcbc-400474ff34de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"
Nov 02 09:52:15.306303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.199919    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f77675-gnslw\" (UID: \"13359001-d62c-44cb-bcbc-400474ff34de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"
Nov 02 09:52:15.306303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.204705    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 02 09:52:15.306303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.204758    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc" podUID="bdbc7509-08cd-45ab-ad9d-32527a7c48af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 02 09:52:15.306303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.305322    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f77675-gnslw\" (UID: \"13359001-d62c-44cb-bcbc-400474ff34de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"
Nov 02 09:52:15.306303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.305413    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f77675-gnslw\" (UID: \"13359001-d62c-44cb-bcbc-400474ff34de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"
Nov 02 09:52:15.306303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.305500    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v25p\" (UniqueName: \"kubernetes.io/projected/13359001-d62c-44cb-bcbc-400474ff34de-kube-api-access-4v25p\") pod \"maas-api-revoke-sub-a3f77675-gnslw\" (UID: \"13359001-d62c-44cb-bcbc-400474ff34de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"
Nov 02 09:52:15.311014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.309074    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f77675-gnslw\" (UID: \"13359001-d62c-44cb-bcbc-400474ff34de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"
Nov 02 09:52:15.337276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.324354    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v25p\" (UniqueName: \"kubernetes.io/projected/13359001-d62c-44cb-bcbc-400474ff34de-kube-api-access-4v25p\") pod \"maas-api-revoke-sub-a3f77675-gnslw\" (UID: \"13359001-d62c-44cb-bcbc-400474ff34de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"
Nov 02 09:52:15.361251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.358695    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f77675-gnslw\" (UID: \"13359001-d62c-44cb-bcbc-400474ff34de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"
Nov 02 09:52:15.361576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.361549    4449 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:52:15.361762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.361736    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:52:15.383313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.383277    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"]
Nov 02 09:52:15.588187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.588095    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"]
Nov 02 09:52:15.667428 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.667124    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"
Nov 02 09:52:15.946299 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:15.945953    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4c342351-wztjj"]
Nov 02 09:52:16.042585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:16.041609    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"]
Nov 02 09:52:16.042585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:16.041879    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj" podUID="1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" containerName="istio-proxy" containerID="cri-o://e9926a181c02240a8d90a9e33ed86299ad4920c90f7af04aae7913a7a427c360" gracePeriod=30
Nov 02 09:52:16.072379 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:16.072334    4449 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.156:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:52:16.093832 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:16.093800    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:52:16.104851 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:16.104662    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:52:16.106687 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:16.106624    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:52:16.173247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:16.163710    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 02 09:52:16.487242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:16.482389    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc" podUID="bdbc7509-08cd-45ab-ad9d-32527a7c48af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 02 09:52:16.487242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:16.269432    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:52:16.632492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:16.632444    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:52:17.343155 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:17.343095    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cba9d18-29a5-4001-8970-eef712944669" path="/var/lib/kubelet/pods/0cba9d18-29a5-4001-8970-eef712944669/volumes"
Nov 02 09:52:17.382765 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:52:17.382690    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13359001_d62c_44cb_bcbc_400474ff34de.slice/crio-0f8c6712a6e683e61c04962e369c960f8f8e9f3a92358ee9ac9ceadcbe3c3096 WatchSource:0}: Error finding container 0f8c6712a6e683e61c04962e369c960f8f8e9f3a92358ee9ac9ceadcbe3c3096: Status 404 returned error can't find the container with id 0f8c6712a6e683e61c04962e369c960f8f8e9f3a92358ee9ac9ceadcbe3c3096
Nov 02 09:52:17.382916 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:17.382846    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"]
Nov 02 09:52:17.417907 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:17.417875    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:17.418871 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:17.418793    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:17.427013 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:17.426960    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:52:17.615611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:17.615552    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859" podUID="a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b" containerName="payload-pre-processing" containerID="cri-o://2f96fff47ffc92d9e231b808597b71f98ac2fde6243044680db6386dcc7b3c62" gracePeriod=25
Nov 02 09:52:17.920802 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:17.920729    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"
Nov 02 09:52:18.018697 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.018664    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-kube-api-access-8cpcw\" (UniqueName: \"kubernetes.io/projected/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-kube-api-access-8cpcw\") pod \"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b\" (UID: \"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b\") "
Nov 02 09:52:18.018872 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.018823    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-plugins-config-volume\") pod \"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b\" (UID: \"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b\") "
Nov 02 09:52:18.019272 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.019246    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-plugins-config-volume" pod "a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b" (UID: "a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:52:18.019492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.019477    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:18.020948 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.020922    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-kube-api-access-8cpcw" pod "a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b" (UID: "a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b"). InnerVolumeSpecName "kube-api-access-8cpcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:18.120907 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.120873    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cpcw\" (UniqueName: \"kubernetes.io/projected/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b-kube-api-access-8cpcw\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:18.270806 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.270715    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw" event={"ID":"13359001-d62c-44cb-bcbc-400474ff34de","Type":"ContainerStarted","Data":"94403f81b1ac88f5bac83a2039d02cd4c6df515067a0d9b2d19cf93cd93ec6f9"}
Nov 02 09:52:18.272654 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.272616    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw" event={"ID":"13359001-d62c-44cb-bcbc-400474ff34de","Type":"ContainerStarted","Data":"0f8c6712a6e683e61c04962e369c960f8f8e9f3a92358ee9ac9ceadcbe3c3096"}
Nov 02 09:52:18.277077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.276132    4449 generic.go:356] "Generic (PLEG): container finished" podID="a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b" containerID="2f96fff47ffc92d9e231b808597b71f98ac2fde6243044680db6386dcc7b3c62" exitCode=0
Nov 02 09:52:18.277077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.276193    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"
Nov 02 09:52:18.277077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.276628    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859" event={"ID":"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b","Type":"ContainerDied","Data":"2f96fff47ffc92d9e231b808597b71f98ac2fde6243044680db6386dcc7b3c62"}
Nov 02 09:52:18.277077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.276678    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859" event={"ID":"a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b","Type":"ContainerDied","Data":"07016e0b2c9e876579f336876a54125ee2712a5d502c44aac4ea7ff3d31b742b"}
Nov 02 09:52:18.277077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.276700    4449 scope.go:122] "RemoveContainer" containerID="2f96fff47ffc92d9e231b808597b71f98ac2fde6243044680db6386dcc7b3c62"
Nov 02 09:52:18.292387 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.292359    4449 scope.go:122] "RemoveContainer" containerID="2f96fff47ffc92d9e231b808597b71f98ac2fde6243044680db6386dcc7b3c62"
Nov 02 09:52:18.292819 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:52:18.292792    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f96fff47ffc92d9e231b808597b71f98ac2fde6243044680db6386dcc7b3c62\": container with ID starting with 2f96fff47ffc92d9e231b808597b71f98ac2fde6243044680db6386dcc7b3c62 not found: ID does not exist" containerID="2f96fff47ffc92d9e231b808597b71f98ac2fde6243044680db6386dcc7b3c62"
Nov 02 09:52:18.292905 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.292832    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f96fff47ffc92d9e231b808597b71f98ac2fde6243044680db6386dcc7b3c62"} err="failed to get container status \"2f96fff47ffc92d9e231b808597b71f98ac2fde6243044680db6386dcc7b3c62\": rpc error: code = NotFound desc = could not find container \"2f96fff47ffc92d9e231b808597b71f98ac2fde6243044680db6386dcc7b3c62\": container with ID starting with 2f96fff47ffc92d9e231b808597b71f98ac2fde6243044680db6386dcc7b3c62 not found: ID does not exist"
Nov 02 09:52:18.296814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.296757    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw" podStartSLOduration=4.296738863 podStartE2EDuration="4.296738863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:18.290901135 +0000 UTC m=+1015.229187912" watchObservedRunningTime="2026-11-02 09:52:18.296738863 +0000 UTC m=+1015.235025640"
Nov 02 09:52:18.310301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.310265    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"]
Nov 02 09:52:18.318891 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:18.318857    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4c342351-7bf9d48b79-hx859"]
Nov 02 09:52:19.032697 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.032662    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"]
Nov 02 09:52:19.038489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.038460    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70a395c3-pxhwb"]
Nov 02 09:52:19.208522 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.208480    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2de4e9a-92da-4f06-a101-94af13ff5099" path="/var/lib/kubelet/pods/a2de4e9a-92da-4f06-a101-94af13ff5099/volumes"
Nov 02 09:52:19.209062 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.209041    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b" path="/var/lib/kubelet/pods/a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b/volumes"
Nov 02 09:52:19.286682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.286652    4449 generic.go:356] "Generic (PLEG): container finished" podID="1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" containerID="e9926a181c02240a8d90a9e33ed86299ad4920c90f7af04aae7913a7a427c360" exitCode=0
Nov 02 09:52:19.286830 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.286725    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj" event={"ID":"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679","Type":"ContainerDied","Data":"e9926a181c02240a8d90a9e33ed86299ad4920c90f7af04aae7913a7a427c360"}
Nov 02 09:52:19.288948 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.288928    4449 generic.go:356] "Generic (PLEG): container finished" podID="13359001-d62c-44cb-bcbc-400474ff34de" containerID="94403f81b1ac88f5bac83a2039d02cd4c6df515067a0d9b2d19cf93cd93ec6f9" exitCode=0
Nov 02 09:52:19.289053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.288997    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw" event={"ID":"13359001-d62c-44cb-bcbc-400474ff34de","Type":"ContainerDied","Data":"94403f81b1ac88f5bac83a2039d02cd4c6df515067a0d9b2d19cf93cd93ec6f9"}
Nov 02 09:52:19.426122 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.426093    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:52:19.545122 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545030    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-socket\") pod \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") "
Nov 02 09:52:19.545122 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545103    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-credential-socket\") pod \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") "
Nov 02 09:52:19.545399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545141    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-data\") pod \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") "
Nov 02 09:52:19.545399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545161    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-podinfo\") pod \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") "
Nov 02 09:52:19.545399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545208    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istiod-ca-cert\") pod \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") "
Nov 02 09:52:19.545399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545338    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-token\" (UniqueName: \"kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-token\") pod \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") "
Nov 02 09:52:19.545399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545383    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-envoy\") pod \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") "
Nov 02 09:52:19.545670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545409    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-socket" pod "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" (UID: "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:19.545670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545496    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-certs\") pod \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") "
Nov 02 09:52:19.545670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545536    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-credential-socket" pod "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" (UID: "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:19.545670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545551    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-data" pod "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" (UID: "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:19.545670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545575    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-kube-api-access-7qmd2\" (UniqueName: \"kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-kube-api-access-7qmd2\") pod \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\" (UID: \"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679\") "
Nov 02 09:52:19.545987 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545809    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-certs" pod "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" (UID: "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:19.546534 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.545731    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istiod-ca-cert" pod "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" (UID: "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:52:19.546534 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.546438    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:19.546534 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.546461    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:19.546534 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.546480    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:19.546534 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.546494    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:19.546534 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.546509    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:19.547791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.547756    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-envoy" pod "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" (UID: "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:19.547895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.547861    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-podinfo" pod "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" (UID: "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:52:19.548290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.548268    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-token" pod "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" (UID: "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:19.548381 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.548298    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-kube-api-access-7qmd2" pod "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" (UID: "1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679"). InnerVolumeSpecName "kube-api-access-7qmd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:19.648147 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.648111    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:19.648147 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.648142    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:19.648147 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.648152    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:19.648415 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:19.648161    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qmd2\" (UniqueName: \"kubernetes.io/projected/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679-kube-api-access-7qmd2\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:20.302137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.302091    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"
Nov 02 09:52:20.302137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.302119    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj" event={"ID":"1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679","Type":"ContainerDied","Data":"e5e494450f4149d332ec4e653ada89c47df1dbdb9dec4f59e14957ffef952894"}
Nov 02 09:52:20.302720 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.302177    4449 scope.go:122] "RemoveContainer" containerID="e9926a181c02240a8d90a9e33ed86299ad4920c90f7af04aae7913a7a427c360"
Nov 02 09:52:20.333819 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.333784    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"]
Nov 02 09:52:20.363752 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.363722    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4c342351-gw-openshift-default-5bd6866449-t8rlj"]
Nov 02 09:52:20.460026 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.459988    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"
Nov 02 09:52:20.563194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.563116    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13359001-d62c-44cb-bcbc-400474ff34de-kube-api-access-4v25p\" (UniqueName: \"kubernetes.io/projected/13359001-d62c-44cb-bcbc-400474ff34de-kube-api-access-4v25p\") pod \"13359001-d62c-44cb-bcbc-400474ff34de\" (UID: \"13359001-d62c-44cb-bcbc-400474ff34de\") "
Nov 02 09:52:20.563194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.563159    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-xks-ca\") pod \"13359001-d62c-44cb-bcbc-400474ff34de\" (UID: \"13359001-d62c-44cb-bcbc-400474ff34de\") "
Nov 02 09:52:20.563194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.563200    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-openshift-ca\") pod \"13359001-d62c-44cb-bcbc-400474ff34de\" (UID: \"13359001-d62c-44cb-bcbc-400474ff34de\") "
Nov 02 09:52:20.563649 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.563623    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-openshift-ca" pod "13359001-d62c-44cb-bcbc-400474ff34de" (UID: "13359001-d62c-44cb-bcbc-400474ff34de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:52:20.563838 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.563819    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:20.565501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.565477    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13359001-d62c-44cb-bcbc-400474ff34de-kube-api-access-4v25p" pod "13359001-d62c-44cb-bcbc-400474ff34de" (UID: "13359001-d62c-44cb-bcbc-400474ff34de"). InnerVolumeSpecName "kube-api-access-4v25p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:20.565622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.565535    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-xks-ca" pod "13359001-d62c-44cb-bcbc-400474ff34de" (UID: "13359001-d62c-44cb-bcbc-400474ff34de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:52:20.664745 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.664717    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v25p\" (UniqueName: \"kubernetes.io/projected/13359001-d62c-44cb-bcbc-400474ff34de-kube-api-access-4v25p\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:20.664745 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:20.664745    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13359001-d62c-44cb-bcbc-400474ff34de-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:21.027794 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.027707    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"]
Nov 02 09:52:21.032814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.032792    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b714b53e-ks7bf"]
Nov 02 09:52:21.207710 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.207682    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" path="/var/lib/kubelet/pods/1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679/volumes"
Nov 02 09:52:21.208209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.208191    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a40410d-eea1-4a9e-a9de-0be97cf4bf1b" path="/var/lib/kubelet/pods/8a40410d-eea1-4a9e-a9de-0be97cf4bf1b/volumes"
Nov 02 09:52:21.314121 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.314080    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw" event={"ID":"13359001-d62c-44cb-bcbc-400474ff34de","Type":"ContainerDied","Data":"0f8c6712a6e683e61c04962e369c960f8f8e9f3a92358ee9ac9ceadcbe3c3096"}
Nov 02 09:52:21.314121 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.314103    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"
Nov 02 09:52:21.314622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.314139    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f8c6712a6e683e61c04962e369c960f8f8e9f3a92358ee9ac9ceadcbe3c3096"
Nov 02 09:52:21.519180 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.519144    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:21.532341 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.532301    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:21.553720 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.553685    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"]
Nov 02 09:52:21.554409 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.554390    4449 cpu_manager.go:414] "containerMap: removing container" podUID="1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" containerName="istio-proxy"
Nov 02 09:52:21.554409 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.554407    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" containerName="istio-proxy"
Nov 02 09:52:21.554595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.554429    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b" containerName="payload-pre-processing"
Nov 02 09:52:21.554595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.554435    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b" containerName="payload-pre-processing"
Nov 02 09:52:21.554595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.554470    4449 cpu_manager.go:414] "containerMap: removing container" podUID="13359001-d62c-44cb-bcbc-400474ff34de" containerName="revoke-keys"
Nov 02 09:52:21.554595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.554475    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13359001-d62c-44cb-bcbc-400474ff34de" containerName="revoke-keys"
Nov 02 09:52:21.554595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.554570    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="13359001-d62c-44cb-bcbc-400474ff34de" containerName="revoke-keys"
Nov 02 09:52:21.554595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.554580    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="1243f9d4-8d6f-4ab0-8a86-bd5ccbf2b679" containerName="istio-proxy"
Nov 02 09:52:21.554595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.554588    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67b2b4d-d0b9-4245-9fba-6b8a9671eb9b" containerName="payload-pre-processing"
Nov 02 09:52:21.558028 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.558005    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"
Nov 02 09:52:21.562128 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.562096    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-62e67514\""
Nov 02 09:52:21.563342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.563319    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-62e67514-dockercfg-r44xl\""
Nov 02 09:52:21.570319 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.570291    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"]
Nov 02 09:52:21.620108 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.620068    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"]
Nov 02 09:52:21.623680 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.623660    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"
Nov 02 09:52:21.638007 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.637978    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"]
Nov 02 09:52:21.706566 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.706514    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s78j6\" (UniqueName: \"kubernetes.io/projected/7ba87092-50fe-4937-9ddb-ab9e88adbf53-kube-api-access-s78j6\") pod \"payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb\" (UID: \"7ba87092-50fe-4937-9ddb-ab9e88adbf53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"
Nov 02 09:52:21.707764 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.707726    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba87092-50fe-4937-9ddb-ab9e88adbf53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb\" (UID: \"7ba87092-50fe-4937-9ddb-ab9e88adbf53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"
Nov 02 09:52:21.810118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.810080    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdjnc\" (UniqueName: \"kubernetes.io/projected/f299fcdd-d631-4f4f-bb82-b32e5349ba76-kube-api-access-mdjnc\") pod \"payload-processing-e2e-ait-62e67514-7b847995-hjrwz\" (UID: \"f299fcdd-d631-4f4f-bb82-b32e5349ba76\") " pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"
Nov 02 09:52:21.810287 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.810188    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba87092-50fe-4937-9ddb-ab9e88adbf53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb\" (UID: \"7ba87092-50fe-4937-9ddb-ab9e88adbf53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"
Nov 02 09:52:21.810434 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.810288    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f299fcdd-d631-4f4f-bb82-b32e5349ba76-plugins-config-volume\") pod \"payload-processing-e2e-ait-62e67514-7b847995-hjrwz\" (UID: \"f299fcdd-d631-4f4f-bb82-b32e5349ba76\") " pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"
Nov 02 09:52:21.810434 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.810372    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78j6\" (UniqueName: \"kubernetes.io/projected/7ba87092-50fe-4937-9ddb-ab9e88adbf53-kube-api-access-s78j6\") pod \"payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb\" (UID: \"7ba87092-50fe-4937-9ddb-ab9e88adbf53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"
Nov 02 09:52:21.811489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.811457    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba87092-50fe-4937-9ddb-ab9e88adbf53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb\" (UID: \"7ba87092-50fe-4937-9ddb-ab9e88adbf53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"
Nov 02 09:52:21.823658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.823593    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78j6\" (UniqueName: \"kubernetes.io/projected/7ba87092-50fe-4937-9ddb-ab9e88adbf53-kube-api-access-s78j6\") pod \"payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb\" (UID: \"7ba87092-50fe-4937-9ddb-ab9e88adbf53\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"
Nov 02 09:52:21.879862 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.879831    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"
Nov 02 09:52:21.911310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.911277    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f299fcdd-d631-4f4f-bb82-b32e5349ba76-plugins-config-volume\") pod \"payload-processing-e2e-ait-62e67514-7b847995-hjrwz\" (UID: \"f299fcdd-d631-4f4f-bb82-b32e5349ba76\") " pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"
Nov 02 09:52:21.911467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.911453    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdjnc\" (UniqueName: \"kubernetes.io/projected/f299fcdd-d631-4f4f-bb82-b32e5349ba76-kube-api-access-mdjnc\") pod \"payload-processing-e2e-ait-62e67514-7b847995-hjrwz\" (UID: \"f299fcdd-d631-4f4f-bb82-b32e5349ba76\") " pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"
Nov 02 09:52:21.912296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.912271    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f299fcdd-d631-4f4f-bb82-b32e5349ba76-plugins-config-volume\") pod \"payload-processing-e2e-ait-62e67514-7b847995-hjrwz\" (UID: \"f299fcdd-d631-4f4f-bb82-b32e5349ba76\") " pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"
Nov 02 09:52:21.925075 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.925047    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdjnc\" (UniqueName: \"kubernetes.io/projected/f299fcdd-d631-4f4f-bb82-b32e5349ba76-kube-api-access-mdjnc\") pod \"payload-processing-e2e-ait-62e67514-7b847995-hjrwz\" (UID: \"f299fcdd-d631-4f4f-bb82-b32e5349ba76\") " pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"
Nov 02 09:52:21.939281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:21.939119    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"
Nov 02 09:52:22.166098 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:52:22.166066    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba87092_50fe_4937_9ddb_ab9e88adbf53.slice/crio-d71eec3a114550a1eb82c18ac467d81b5c31b27742fe4a416887ea6c55ce911a WatchSource:0}: Error finding container d71eec3a114550a1eb82c18ac467d81b5c31b27742fe4a416887ea6c55ce911a: Status 404 returned error can't find the container with id d71eec3a114550a1eb82c18ac467d81b5c31b27742fe4a416887ea6c55ce911a
Nov 02 09:52:22.167915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:22.167846    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"]
Nov 02 09:52:22.242024 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:52:22.241997    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf299fcdd_d631_4f4f_bb82_b32e5349ba76.slice/crio-69124f0dc78e96467679b952ac6d08c1fb52f2818ff6b6720b75599c5f34cbdc WatchSource:0}: Error finding container 69124f0dc78e96467679b952ac6d08c1fb52f2818ff6b6720b75599c5f34cbdc: Status 404 returned error can't find the container with id 69124f0dc78e96467679b952ac6d08c1fb52f2818ff6b6720b75599c5f34cbdc
Nov 02 09:52:22.243836 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:22.243815    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"]
Nov 02 09:52:22.324658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:22.324624    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb" event={"ID":"7ba87092-50fe-4937-9ddb-ab9e88adbf53","Type":"ContainerStarted","Data":"03bc7af86fafc291aadeeb1627c5e45b45603e7c7cb1c67379af6610b10adca3"}
Nov 02 09:52:22.325157 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:22.324684    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb" event={"ID":"7ba87092-50fe-4937-9ddb-ab9e88adbf53","Type":"ContainerStarted","Data":"d71eec3a114550a1eb82c18ac467d81b5c31b27742fe4a416887ea6c55ce911a"}
Nov 02 09:52:22.325157 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:22.324712    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"
Nov 02 09:52:22.326676 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:22.326642    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz" event={"ID":"f299fcdd-d631-4f4f-bb82-b32e5349ba76","Type":"ContainerStarted","Data":"64aaf8ad4f59063e3cf8f4eb805bd5a012347969bb9654a844eddd1f8b7c60c3"}
Nov 02 09:52:22.326805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:22.326700    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz" event={"ID":"f299fcdd-d631-4f4f-bb82-b32e5349ba76","Type":"ContainerStarted","Data":"69124f0dc78e96467679b952ac6d08c1fb52f2818ff6b6720b75599c5f34cbdc"}
Nov 02 09:52:22.326805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:22.326742    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"
Nov 02 09:52:22.338254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:22.338207    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:52:22.347446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:22.347393    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb" podStartSLOduration=1.347357128 podStartE2EDuration="1.347357128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:22.343961647 +0000 UTC m=+1019.282248419" watchObservedRunningTime="2026-11-02 09:52:22.347357128 +0000 UTC m=+1019.285643903"
Nov 02 09:52:22.389318 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:22.389255    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz" podStartSLOduration=1.389232015 podStartE2EDuration="1.389232015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:22.381950046 +0000 UTC m=+1019.320236843" watchObservedRunningTime="2026-11-02 09:52:22.389232015 +0000 UTC m=+1019.327518790"
Nov 02 09:52:23.715840 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:23.715787    4449 scope.go:122] "RemoveContainer" containerID="5e12f7b82a63d2d1dd12fbae2b20650fae5634af4d86024cb26bbea4e495ec72"
Nov 02 09:52:31.453367 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.453335    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"]
Nov 02 09:52:31.458773 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.458752    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"
Nov 02 09:52:31.463525 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.463495    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:52:31.463703 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.463523    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:52:31.463703 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.463523    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:52:31.463703 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.463570    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:52:31.466777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.466749    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"]
Nov 02 09:52:31.533047 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.533014    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba81329d-6lbzx\" (UID: \"782c8ffc-52ff-467b-a380-cc2d4e77c952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"
Nov 02 09:52:31.533255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.533096    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzcd6\" (UniqueName: \"kubernetes.io/projected/782c8ffc-52ff-467b-a380-cc2d4e77c952-kube-api-access-gzcd6\") pod \"maas-api-revoke-sub-ba81329d-6lbzx\" (UID: \"782c8ffc-52ff-467b-a380-cc2d4e77c952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"
Nov 02 09:52:31.533255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.533171    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba81329d-6lbzx\" (UID: \"782c8ffc-52ff-467b-a380-cc2d4e77c952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"
Nov 02 09:52:31.634971 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.634933    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba81329d-6lbzx\" (UID: \"782c8ffc-52ff-467b-a380-cc2d4e77c952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"
Nov 02 09:52:31.635190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.634997    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzcd6\" (UniqueName: \"kubernetes.io/projected/782c8ffc-52ff-467b-a380-cc2d4e77c952-kube-api-access-gzcd6\") pod \"maas-api-revoke-sub-ba81329d-6lbzx\" (UID: \"782c8ffc-52ff-467b-a380-cc2d4e77c952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"
Nov 02 09:52:31.635190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.635034    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba81329d-6lbzx\" (UID: \"782c8ffc-52ff-467b-a380-cc2d4e77c952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"
Nov 02 09:52:31.635636 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.635612    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba81329d-6lbzx\" (UID: \"782c8ffc-52ff-467b-a380-cc2d4e77c952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"
Nov 02 09:52:31.637626 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.637602    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba81329d-6lbzx\" (UID: \"782c8ffc-52ff-467b-a380-cc2d4e77c952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"
Nov 02 09:52:31.648141 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.648101    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzcd6\" (UniqueName: \"kubernetes.io/projected/782c8ffc-52ff-467b-a380-cc2d4e77c952-kube-api-access-gzcd6\") pod \"maas-api-revoke-sub-ba81329d-6lbzx\" (UID: \"782c8ffc-52ff-467b-a380-cc2d4e77c952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"
Nov 02 09:52:31.776374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:31.776292    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"
Nov 02 09:52:32.061715 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:52:32.061676    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782c8ffc_52ff_467b_a380_cc2d4e77c952.slice/crio-c2f3ee387f473b4f900998f1f8dc5118700fe3857146c545d38a089f7932f8e4 WatchSource:0}: Error finding container c2f3ee387f473b4f900998f1f8dc5118700fe3857146c545d38a089f7932f8e4: Status 404 returned error can't find the container with id c2f3ee387f473b4f900998f1f8dc5118700fe3857146c545d38a089f7932f8e4
Nov 02 09:52:32.063300 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:32.063272    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"]
Nov 02 09:52:32.501778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:32.501737    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx" event={"ID":"782c8ffc-52ff-467b-a380-cc2d4e77c952","Type":"ContainerStarted","Data":"a2c271eb1e8fcd8fdf1207532627d7da3d304d9c8fa8ba1f1440e99bb019a59f"}
Nov 02 09:52:32.502171 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:32.501801    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx" event={"ID":"782c8ffc-52ff-467b-a380-cc2d4e77c952","Type":"ContainerStarted","Data":"c2f3ee387f473b4f900998f1f8dc5118700fe3857146c545d38a089f7932f8e4"}
Nov 02 09:52:32.521348 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:32.521291    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx" podStartSLOduration=1.521277029 podStartE2EDuration="1.521277029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:32.519382952 +0000 UTC m=+1029.457669753" watchObservedRunningTime="2026-11-02 09:52:32.521277029 +0000 UTC m=+1029.459563805"
Nov 02 09:52:33.031249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:33.031176    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"]
Nov 02 09:52:33.036543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:33.036522    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a42c0e35-x2c6f"]
Nov 02 09:52:33.209386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:33.209349    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41246da8-0317-4f59-a90e-6b4f02b28115" path="/var/lib/kubelet/pods/41246da8-0317-4f59-a90e-6b4f02b28115/volumes"
Nov 02 09:52:33.357588 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:33.357560    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"
Nov 02 09:52:33.357752 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:33.357630    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"
Nov 02 09:52:33.523359 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:33.523323    4449 generic.go:356] "Generic (PLEG): container finished" podID="782c8ffc-52ff-467b-a380-cc2d4e77c952" containerID="a2c271eb1e8fcd8fdf1207532627d7da3d304d9c8fa8ba1f1440e99bb019a59f" exitCode=0
Nov 02 09:52:33.523730 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:33.523370    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx" event={"ID":"782c8ffc-52ff-467b-a380-cc2d4e77c952","Type":"ContainerDied","Data":"a2c271eb1e8fcd8fdf1207532627d7da3d304d9c8fa8ba1f1440e99bb019a59f"}
Nov 02 09:52:34.703526 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:34.703499    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"
Nov 02 09:52:34.883756 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:34.883719    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-xks-ca\") pod \"782c8ffc-52ff-467b-a380-cc2d4e77c952\" (UID: \"782c8ffc-52ff-467b-a380-cc2d4e77c952\") "
Nov 02 09:52:34.883756 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:34.883761    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782c8ffc-52ff-467b-a380-cc2d4e77c952-kube-api-access-gzcd6\" (UniqueName: \"kubernetes.io/projected/782c8ffc-52ff-467b-a380-cc2d4e77c952-kube-api-access-gzcd6\") pod \"782c8ffc-52ff-467b-a380-cc2d4e77c952\" (UID: \"782c8ffc-52ff-467b-a380-cc2d4e77c952\") "
Nov 02 09:52:34.884010 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:34.883923    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-openshift-ca\") pod \"782c8ffc-52ff-467b-a380-cc2d4e77c952\" (UID: \"782c8ffc-52ff-467b-a380-cc2d4e77c952\") "
Nov 02 09:52:34.884335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:34.884306    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-openshift-ca" pod "782c8ffc-52ff-467b-a380-cc2d4e77c952" (UID: "782c8ffc-52ff-467b-a380-cc2d4e77c952"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:52:34.886108 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:34.886070    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-xks-ca" pod "782c8ffc-52ff-467b-a380-cc2d4e77c952" (UID: "782c8ffc-52ff-467b-a380-cc2d4e77c952"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:52:34.886300 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:34.886278    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782c8ffc-52ff-467b-a380-cc2d4e77c952-kube-api-access-gzcd6" pod "782c8ffc-52ff-467b-a380-cc2d4e77c952" (UID: "782c8ffc-52ff-467b-a380-cc2d4e77c952"). InnerVolumeSpecName "kube-api-access-gzcd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:34.989363 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:34.988693    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:34.989363 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:34.988735    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/782c8ffc-52ff-467b-a380-cc2d4e77c952-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:34.989363 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:34.988748    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzcd6\" (UniqueName: \"kubernetes.io/projected/782c8ffc-52ff-467b-a380-cc2d4e77c952-kube-api-access-gzcd6\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:35.543642 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:35.543603    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx" event={"ID":"782c8ffc-52ff-467b-a380-cc2d4e77c952","Type":"ContainerDied","Data":"c2f3ee387f473b4f900998f1f8dc5118700fe3857146c545d38a089f7932f8e4"}
Nov 02 09:52:35.543642 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:35.543625    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"
Nov 02 09:52:35.543858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:35.543658    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2f3ee387f473b4f900998f1f8dc5118700fe3857146c545d38a089f7932f8e4"
Nov 02 09:52:36.555382 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:52:36.555348    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:52:42.027475 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:42.027439    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"]
Nov 02 09:52:42.033024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:42.032991    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa69fa05-snnv7"]
Nov 02 09:52:43.209312 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:43.209280    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef689a32-1555-4d9f-93d9-9c94fdb1d8f0" path="/var/lib/kubelet/pods/ef689a32-1555-4d9f-93d9-9c94fdb1d8f0/volumes"
Nov 02 09:52:43.994717 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:43.994683    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"]
Nov 02 09:52:43.995414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:43.995399    4449 cpu_manager.go:414] "containerMap: removing container" podUID="782c8ffc-52ff-467b-a380-cc2d4e77c952" containerName="revoke-keys"
Nov 02 09:52:43.995482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:43.995416    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782c8ffc-52ff-467b-a380-cc2d4e77c952" containerName="revoke-keys"
Nov 02 09:52:43.995556 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:43.995547    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="782c8ffc-52ff-467b-a380-cc2d4e77c952" containerName="revoke-keys"
Nov 02 09:52:43.999715 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:43.999700    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"
Nov 02 09:52:44.004002 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.003973    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:52:44.004875 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.004818    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:52:44.004875 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.004828    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:52:44.005115 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.004905    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:52:44.008605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.008571    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"]
Nov 02 09:52:44.036168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.036137    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"]
Nov 02 09:52:44.044764 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.044719    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb14f2e1-vnqkk"]
Nov 02 09:52:44.110244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.109330    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4605dd-94gfr\" (UID: \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"
Nov 02 09:52:44.110244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.109393    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4605dd-94gfr\" (UID: \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"
Nov 02 09:52:44.110244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.109513    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcxz8\" (UniqueName: \"kubernetes.io/projected/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-kube-api-access-rcxz8\") pod \"maas-api-revoke-sub-0b4605dd-94gfr\" (UID: \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"
Nov 02 09:52:44.210653 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.210617    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcxz8\" (UniqueName: \"kubernetes.io/projected/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-kube-api-access-rcxz8\") pod \"maas-api-revoke-sub-0b4605dd-94gfr\" (UID: \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"
Nov 02 09:52:44.211118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.210885    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4605dd-94gfr\" (UID: \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"
Nov 02 09:52:44.211118 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.210928    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4605dd-94gfr\" (UID: \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"
Nov 02 09:52:44.211860 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.211824    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4605dd-94gfr\" (UID: \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"
Nov 02 09:52:44.213643 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.213614    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4605dd-94gfr\" (UID: \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"
Nov 02 09:52:44.226512 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.226479    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcxz8\" (UniqueName: \"kubernetes.io/projected/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-kube-api-access-rcxz8\") pod \"maas-api-revoke-sub-0b4605dd-94gfr\" (UID: \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"
Nov 02 09:52:44.319819 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.319789    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"
Nov 02 09:52:44.607377 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:52:44.607344    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf96d9a_aa33_4614_9a3b_1f4d90ada054.slice/crio-e6cb7e72fa487ba2429061c234a5326950851bbfdc5bccf20142609db1207c6b WatchSource:0}: Error finding container e6cb7e72fa487ba2429061c234a5326950851bbfdc5bccf20142609db1207c6b: Status 404 returned error can't find the container with id e6cb7e72fa487ba2429061c234a5326950851bbfdc5bccf20142609db1207c6b
Nov 02 09:52:44.608623 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.608596    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"]
Nov 02 09:52:44.679872 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:44.679836    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr" event={"ID":"cdf96d9a-aa33-4614-9a3b-1f4d90ada054","Type":"ContainerStarted","Data":"e6cb7e72fa487ba2429061c234a5326950851bbfdc5bccf20142609db1207c6b"}
Nov 02 09:52:45.209469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:45.209383    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="226bd2c7-0279-4dad-8b4a-512d5f30588e" path="/var/lib/kubelet/pods/226bd2c7-0279-4dad-8b4a-512d5f30588e/volumes"
Nov 02 09:52:45.690203 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:45.690171    4449 generic.go:356] "Generic (PLEG): container finished" podID="cdf96d9a-aa33-4614-9a3b-1f4d90ada054" containerID="0e927a52694b31b7217598e908d5c9c1855983a7d96bd4ecf34ef0e2bc8b8a10" exitCode=0
Nov 02 09:52:45.690611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:45.690250    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr" event={"ID":"cdf96d9a-aa33-4614-9a3b-1f4d90ada054","Type":"ContainerDied","Data":"0e927a52694b31b7217598e908d5c9c1855983a7d96bd4ecf34ef0e2bc8b8a10"}
Nov 02 09:52:46.613361 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.613317    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"]
Nov 02 09:52:46.633606 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.633540    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"
Nov 02 09:52:46.635806 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.635778    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"]
Nov 02 09:52:46.747960 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.747920    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-62e67514-ld7gp\" (UID: \"ed888c29-c7e6-45c4-9f76-03ddf2051471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"
Nov 02 09:52:46.748483 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.748012    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cd58\" (UniqueName: \"kubernetes.io/projected/ed888c29-c7e6-45c4-9f76-03ddf2051471-kube-api-access-6cd58\") pod \"maas-api-revoke-keys-e2e-ait-62e67514-ld7gp\" (UID: \"ed888c29-c7e6-45c4-9f76-03ddf2051471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"
Nov 02 09:52:46.748483 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.748126    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-62e67514-ld7gp\" (UID: \"ed888c29-c7e6-45c4-9f76-03ddf2051471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"
Nov 02 09:52:46.849894 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.849858    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-62e67514-ld7gp\" (UID: \"ed888c29-c7e6-45c4-9f76-03ddf2051471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"
Nov 02 09:52:46.850088 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.849967    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cd58\" (UniqueName: \"kubernetes.io/projected/ed888c29-c7e6-45c4-9f76-03ddf2051471-kube-api-access-6cd58\") pod \"maas-api-revoke-keys-e2e-ait-62e67514-ld7gp\" (UID: \"ed888c29-c7e6-45c4-9f76-03ddf2051471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"
Nov 02 09:52:46.850088 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.850015    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-62e67514-ld7gp\" (UID: \"ed888c29-c7e6-45c4-9f76-03ddf2051471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"
Nov 02 09:52:46.850786 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.850761    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-62e67514-ld7gp\" (UID: \"ed888c29-c7e6-45c4-9f76-03ddf2051471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"
Nov 02 09:52:46.852584 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.852562    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-62e67514-ld7gp\" (UID: \"ed888c29-c7e6-45c4-9f76-03ddf2051471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"
Nov 02 09:52:46.864185 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.864106    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cd58\" (UniqueName: \"kubernetes.io/projected/ed888c29-c7e6-45c4-9f76-03ddf2051471-kube-api-access-6cd58\") pod \"maas-api-revoke-keys-e2e-ait-62e67514-ld7gp\" (UID: \"ed888c29-c7e6-45c4-9f76-03ddf2051471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"
Nov 02 09:52:46.892280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.892206    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"
Nov 02 09:52:46.951331 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.951293    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-kube-api-access-rcxz8\" (UniqueName: \"kubernetes.io/projected/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-kube-api-access-rcxz8\") pod \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\" (UID: \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\") "
Nov 02 09:52:46.951538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.951453    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-openshift-ca\") pod \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\" (UID: \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\") "
Nov 02 09:52:46.951538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.951512    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-xks-ca\") pod \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\" (UID: \"cdf96d9a-aa33-4614-9a3b-1f4d90ada054\") "
Nov 02 09:52:46.951935 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.951900    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-openshift-ca" pod "cdf96d9a-aa33-4614-9a3b-1f4d90ada054" (UID: "cdf96d9a-aa33-4614-9a3b-1f4d90ada054"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:52:46.952274 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.952208    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:46.954067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.954034    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-xks-ca" pod "cdf96d9a-aa33-4614-9a3b-1f4d90ada054" (UID: "cdf96d9a-aa33-4614-9a3b-1f4d90ada054"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:52:46.954188 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.954117    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"
Nov 02 09:52:46.954441 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:46.954408    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-kube-api-access-rcxz8" pod "cdf96d9a-aa33-4614-9a3b-1f4d90ada054" (UID: "cdf96d9a-aa33-4614-9a3b-1f4d90ada054"). InnerVolumeSpecName "kube-api-access-rcxz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:47.053830 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:47.053798    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcxz8\" (UniqueName: \"kubernetes.io/projected/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-kube-api-access-rcxz8\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:47.053830 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:47.053829    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf96d9a-aa33-4614-9a3b-1f4d90ada054-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:47.248093 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:52:47.248054    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded888c29_c7e6_45c4_9f76_03ddf2051471.slice/crio-a32522a6bf4399b4d793b1eaf3ef9b80950867ed01c06500b9fcc55da22f7a57 WatchSource:0}: Error finding container a32522a6bf4399b4d793b1eaf3ef9b80950867ed01c06500b9fcc55da22f7a57: Status 404 returned error can't find the container with id a32522a6bf4399b4d793b1eaf3ef9b80950867ed01c06500b9fcc55da22f7a57
Nov 02 09:52:47.249057 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:47.249036    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"]
Nov 02 09:52:47.712209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:47.712183    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"
Nov 02 09:52:47.712367 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:47.712247    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr" event={"ID":"cdf96d9a-aa33-4614-9a3b-1f4d90ada054","Type":"ContainerDied","Data":"e6cb7e72fa487ba2429061c234a5326950851bbfdc5bccf20142609db1207c6b"}
Nov 02 09:52:47.712367 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:47.712289    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6cb7e72fa487ba2429061c234a5326950851bbfdc5bccf20142609db1207c6b"
Nov 02 09:52:47.714414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:47.714386    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp" event={"ID":"ed888c29-c7e6-45c4-9f76-03ddf2051471","Type":"ContainerStarted","Data":"65c1a932eb3d1e1e120d8348d3403ea6fba1f7733de9f769752808205d89ba13"}
Nov 02 09:52:47.714545 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:47.714444    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp" event={"ID":"ed888c29-c7e6-45c4-9f76-03ddf2051471","Type":"ContainerStarted","Data":"a32522a6bf4399b4d793b1eaf3ef9b80950867ed01c06500b9fcc55da22f7a57"}
Nov 02 09:52:47.733461 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:47.733411    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp" podStartSLOduration=1.7333973679999999 podStartE2EDuration="1.733397368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:47.731824531 +0000 UTC m=+1044.670111325" watchObservedRunningTime="2026-11-02 09:52:47.733397368 +0000 UTC m=+1044.671684144"
Nov 02 09:52:48.728096 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:48.728058    4449 generic.go:356] "Generic (PLEG): container finished" podID="ed888c29-c7e6-45c4-9f76-03ddf2051471" containerID="65c1a932eb3d1e1e120d8348d3403ea6fba1f7733de9f769752808205d89ba13" exitCode=0
Nov 02 09:52:48.728568 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:48.728097    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp" event={"ID":"ed888c29-c7e6-45c4-9f76-03ddf2051471","Type":"ContainerDied","Data":"65c1a932eb3d1e1e120d8348d3403ea6fba1f7733de9f769752808205d89ba13"}
Nov 02 09:52:49.938161 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:49.938135    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"
Nov 02 09:52:50.096052 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:50.096014    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-openshift-ca\") pod \"ed888c29-c7e6-45c4-9f76-03ddf2051471\" (UID: \"ed888c29-c7e6-45c4-9f76-03ddf2051471\") "
Nov 02 09:52:50.096210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:50.096168    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-xks-ca\") pod \"ed888c29-c7e6-45c4-9f76-03ddf2051471\" (UID: \"ed888c29-c7e6-45c4-9f76-03ddf2051471\") "
Nov 02 09:52:50.096210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:50.096204    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed888c29-c7e6-45c4-9f76-03ddf2051471-kube-api-access-6cd58\" (UniqueName: \"kubernetes.io/projected/ed888c29-c7e6-45c4-9f76-03ddf2051471-kube-api-access-6cd58\") pod \"ed888c29-c7e6-45c4-9f76-03ddf2051471\" (UID: \"ed888c29-c7e6-45c4-9f76-03ddf2051471\") "
Nov 02 09:52:50.096452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:50.096421    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-openshift-ca" pod "ed888c29-c7e6-45c4-9f76-03ddf2051471" (UID: "ed888c29-c7e6-45c4-9f76-03ddf2051471"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:52:50.096744 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:50.096726    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:50.098644 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:50.098610    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-xks-ca" pod "ed888c29-c7e6-45c4-9f76-03ddf2051471" (UID: "ed888c29-c7e6-45c4-9f76-03ddf2051471"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:52:50.098750 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:50.098671    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed888c29-c7e6-45c4-9f76-03ddf2051471-kube-api-access-6cd58" pod "ed888c29-c7e6-45c4-9f76-03ddf2051471" (UID: "ed888c29-c7e6-45c4-9f76-03ddf2051471"). InnerVolumeSpecName "kube-api-access-6cd58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:50.198647 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:50.198607    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed888c29-c7e6-45c4-9f76-03ddf2051471-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:50.198647 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:50.198642    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cd58\" (UniqueName: \"kubernetes.io/projected/ed888c29-c7e6-45c4-9f76-03ddf2051471-kube-api-access-6cd58\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:50.748518 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:50.748487    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"
Nov 02 09:52:50.748518 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:50.748503    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp" event={"ID":"ed888c29-c7e6-45c4-9f76-03ddf2051471","Type":"ContainerDied","Data":"a32522a6bf4399b4d793b1eaf3ef9b80950867ed01c06500b9fcc55da22f7a57"}
Nov 02 09:52:50.748758 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:50.748566    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a32522a6bf4399b4d793b1eaf3ef9b80950867ed01c06500b9fcc55da22f7a57"
Nov 02 09:52:55.077058 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:55.077022    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"]
Nov 02 09:52:55.082457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:55.082425    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54790839-bmgbr"]
Nov 02 09:52:55.207324 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:55.207292    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08edce8a-6a15-4062-83a4-977652f312c8" path="/var/lib/kubelet/pods/08edce8a-6a15-4062-83a4-977652f312c8/volumes"
Nov 02 09:52:56.799080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.799049    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"]
Nov 02 09:52:56.810662 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.810633    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"]
Nov 02 09:52:56.900253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.900198    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"]
Nov 02 09:52:56.901265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.901243    4449 cpu_manager.go:414] "containerMap: removing container" podUID="cdf96d9a-aa33-4614-9a3b-1f4d90ada054" containerName="revoke-keys"
Nov 02 09:52:56.901316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.901267    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf96d9a-aa33-4614-9a3b-1f4d90ada054" containerName="revoke-keys"
Nov 02 09:52:56.901353 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.901335    4449 cpu_manager.go:414] "containerMap: removing container" podUID="ed888c29-c7e6-45c4-9f76-03ddf2051471" containerName="revoke-keys"
Nov 02 09:52:56.901353 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.901343    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed888c29-c7e6-45c4-9f76-03ddf2051471" containerName="revoke-keys"
Nov 02 09:52:56.901524 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.901512    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf96d9a-aa33-4614-9a3b-1f4d90ada054" containerName="revoke-keys"
Nov 02 09:52:56.901570 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.901539    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed888c29-c7e6-45c4-9f76-03ddf2051471" containerName="revoke-keys"
Nov 02 09:52:56.907506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.907475    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"
Nov 02 09:52:56.910592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.910566    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"]
Nov 02 09:52:56.911447 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.911423    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:52:56.911447 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.911436    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:52:56.911447 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.911423    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:52:56.911447 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.911447    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:52:56.999912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.999876    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1bd7cc1-94rdh\" (UID: \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"
Nov 02 09:52:57.000087 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:56.999931    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbm9n\" (UniqueName: \"kubernetes.io/projected/b4d02db6-2385-4eb2-98ce-fae2c070af0c-kube-api-access-fbm9n\") pod \"maas-api-revoke-sub-e1bd7cc1-94rdh\" (UID: \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"
Nov 02 09:52:57.000165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:57.000144    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1bd7cc1-94rdh\" (UID: \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"
Nov 02 09:52:57.106317 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:57.104855    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1bd7cc1-94rdh\" (UID: \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"
Nov 02 09:52:57.106317 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:57.105047    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1bd7cc1-94rdh\" (UID: \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"
Nov 02 09:52:57.106317 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:57.105103    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbm9n\" (UniqueName: \"kubernetes.io/projected/b4d02db6-2385-4eb2-98ce-fae2c070af0c-kube-api-access-fbm9n\") pod \"maas-api-revoke-sub-e1bd7cc1-94rdh\" (UID: \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"
Nov 02 09:52:57.114392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:57.111069    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1bd7cc1-94rdh\" (UID: \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"
Nov 02 09:52:57.156240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:57.152292    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1bd7cc1-94rdh\" (UID: \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"
Nov 02 09:52:57.156240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:57.153487    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbm9n\" (UniqueName: \"kubernetes.io/projected/b4d02db6-2385-4eb2-98ce-fae2c070af0c-kube-api-access-fbm9n\") pod \"maas-api-revoke-sub-e1bd7cc1-94rdh\" (UID: \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"
Nov 02 09:52:57.232935 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:57.232901    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"
Nov 02 09:52:57.518973 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:57.518942    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"]
Nov 02 09:52:57.519087 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:52:57.518991    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d02db6_2385_4eb2_98ce_fae2c070af0c.slice/crio-64ccc4da9c0cacb91581c86f01f697cf035e44dad865825d9f35a0227a05037c WatchSource:0}: Error finding container 64ccc4da9c0cacb91581c86f01f697cf035e44dad865825d9f35a0227a05037c: Status 404 returned error can't find the container with id 64ccc4da9c0cacb91581c86f01f697cf035e44dad865825d9f35a0227a05037c
Nov 02 09:52:57.863009 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:57.862975    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh" event={"ID":"b4d02db6-2385-4eb2-98ce-fae2c070af0c","Type":"ContainerStarted","Data":"64ccc4da9c0cacb91581c86f01f697cf035e44dad865825d9f35a0227a05037c"}
Nov 02 09:52:58.875025 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:58.874925    4449 generic.go:356] "Generic (PLEG): container finished" podID="b4d02db6-2385-4eb2-98ce-fae2c070af0c" containerID="71e063efbf0e65e960223db68d239dafb8482c6624e0153cd194b901b34df338" exitCode=0
Nov 02 09:52:58.875489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:58.875016    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh" event={"ID":"b4d02db6-2385-4eb2-98ce-fae2c070af0c","Type":"ContainerDied","Data":"71e063efbf0e65e960223db68d239dafb8482c6624e0153cd194b901b34df338"}
Nov 02 09:52:59.972862 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:59.972812    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"]
Nov 02 09:52:59.980146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:52:59.980094    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62e67514-ld7gp"]
Nov 02 09:53:00.166247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:00.166179    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"
Nov 02 09:53:00.260781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:00.260744    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d02db6-2385-4eb2-98ce-fae2c070af0c-kube-api-access-fbm9n\" (UniqueName: \"kubernetes.io/projected/b4d02db6-2385-4eb2-98ce-fae2c070af0c-kube-api-access-fbm9n\") pod \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\" (UID: \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\") "
Nov 02 09:53:00.261833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:00.261808    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-xks-ca\") pod \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\" (UID: \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\") "
Nov 02 09:53:00.263150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:00.262458    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-openshift-ca\") pod \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\" (UID: \"b4d02db6-2385-4eb2-98ce-fae2c070af0c\") "
Nov 02 09:53:00.263150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:00.262681    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-openshift-ca" pod "b4d02db6-2385-4eb2-98ce-fae2c070af0c" (UID: "b4d02db6-2385-4eb2-98ce-fae2c070af0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:00.263150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:00.263105    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:00.278435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:00.278390    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-xks-ca" pod "b4d02db6-2385-4eb2-98ce-fae2c070af0c" (UID: "b4d02db6-2385-4eb2-98ce-fae2c070af0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:00.278435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:00.278430    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d02db6-2385-4eb2-98ce-fae2c070af0c-kube-api-access-fbm9n" pod "b4d02db6-2385-4eb2-98ce-fae2c070af0c" (UID: "b4d02db6-2385-4eb2-98ce-fae2c070af0c"). InnerVolumeSpecName "kube-api-access-fbm9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:00.364648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:00.364590    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbm9n\" (UniqueName: \"kubernetes.io/projected/b4d02db6-2385-4eb2-98ce-fae2c070af0c-kube-api-access-fbm9n\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:00.364648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:00.364645    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d02db6-2385-4eb2-98ce-fae2c070af0c-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:00.940241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:00.940105    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh" event={"ID":"b4d02db6-2385-4eb2-98ce-fae2c070af0c","Type":"ContainerDied","Data":"64ccc4da9c0cacb91581c86f01f697cf035e44dad865825d9f35a0227a05037c"}
Nov 02 09:53:00.940241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:00.940156    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64ccc4da9c0cacb91581c86f01f697cf035e44dad865825d9f35a0227a05037c"
Nov 02 09:53:00.940241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:00.940209    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"
Nov 02 09:53:01.236419 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:01.236326    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed888c29-c7e6-45c4-9f76-03ddf2051471" path="/var/lib/kubelet/pods/ed888c29-c7e6-45c4-9f76-03ddf2051471/volumes"
Nov 02 09:53:01.801276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:01.801087    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz" podUID="f299fcdd-d631-4f4f-bb82-b32e5349ba76" containerName="payload-processing" containerID="cri-o://64aaf8ad4f59063e3cf8f4eb805bd5a012347969bb9654a844eddd1f8b7c60c3" gracePeriod=25
Nov 02 09:53:01.816257 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:01.814503    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb" podUID="7ba87092-50fe-4937-9ddb-ab9e88adbf53" containerName="payload-pre-processing" containerID="cri-o://03bc7af86fafc291aadeeb1627c5e45b45603e7c7cb1c67379af6610b10adca3" gracePeriod=25
Nov 02 09:53:01.955263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:01.955175    4449 generic.go:356] "Generic (PLEG): container finished" podID="7ba87092-50fe-4937-9ddb-ab9e88adbf53" containerID="03bc7af86fafc291aadeeb1627c5e45b45603e7c7cb1c67379af6610b10adca3" exitCode=0
Nov 02 09:53:01.955412 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:01.955259    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb" event={"ID":"7ba87092-50fe-4937-9ddb-ab9e88adbf53","Type":"ContainerDied","Data":"03bc7af86fafc291aadeeb1627c5e45b45603e7c7cb1c67379af6610b10adca3"}
Nov 02 09:53:01.958414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:01.958357    4449 generic.go:356] "Generic (PLEG): container finished" podID="f299fcdd-d631-4f4f-bb82-b32e5349ba76" containerID="64aaf8ad4f59063e3cf8f4eb805bd5a012347969bb9654a844eddd1f8b7c60c3" exitCode=0
Nov 02 09:53:01.958561 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:01.958459    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz" event={"ID":"f299fcdd-d631-4f4f-bb82-b32e5349ba76","Type":"ContainerDied","Data":"64aaf8ad4f59063e3cf8f4eb805bd5a012347969bb9654a844eddd1f8b7c60c3"}
Nov 02 09:53:02.142056 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.142030    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"
Nov 02 09:53:02.165947 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.165918    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f299fcdd-d631-4f4f-bb82-b32e5349ba76-kube-api-access-mdjnc\" (UniqueName: \"kubernetes.io/projected/f299fcdd-d631-4f4f-bb82-b32e5349ba76-kube-api-access-mdjnc\") pod \"f299fcdd-d631-4f4f-bb82-b32e5349ba76\" (UID: \"f299fcdd-d631-4f4f-bb82-b32e5349ba76\") "
Nov 02 09:53:02.166115 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.166105    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f299fcdd-d631-4f4f-bb82-b32e5349ba76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f299fcdd-d631-4f4f-bb82-b32e5349ba76-plugins-config-volume\") pod \"f299fcdd-d631-4f4f-bb82-b32e5349ba76\" (UID: \"f299fcdd-d631-4f4f-bb82-b32e5349ba76\") "
Nov 02 09:53:02.166699 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.166668    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f299fcdd-d631-4f4f-bb82-b32e5349ba76-plugins-config-volume" pod "f299fcdd-d631-4f4f-bb82-b32e5349ba76" (UID: "f299fcdd-d631-4f4f-bb82-b32e5349ba76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:02.168670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.168641    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f299fcdd-d631-4f4f-bb82-b32e5349ba76-kube-api-access-mdjnc" pod "f299fcdd-d631-4f4f-bb82-b32e5349ba76" (UID: "f299fcdd-d631-4f4f-bb82-b32e5349ba76"). InnerVolumeSpecName "kube-api-access-mdjnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:02.170545 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.170517    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"
Nov 02 09:53:02.267533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.267501    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba87092-50fe-4937-9ddb-ab9e88adbf53-kube-api-access-s78j6\" (UniqueName: \"kubernetes.io/projected/7ba87092-50fe-4937-9ddb-ab9e88adbf53-kube-api-access-s78j6\") pod \"7ba87092-50fe-4937-9ddb-ab9e88adbf53\" (UID: \"7ba87092-50fe-4937-9ddb-ab9e88adbf53\") "
Nov 02 09:53:02.267934 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.267664    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba87092-50fe-4937-9ddb-ab9e88adbf53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba87092-50fe-4937-9ddb-ab9e88adbf53-plugins-config-volume\") pod \"7ba87092-50fe-4937-9ddb-ab9e88adbf53\" (UID: \"7ba87092-50fe-4937-9ddb-ab9e88adbf53\") "
Nov 02 09:53:02.268108 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.268083    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba87092-50fe-4937-9ddb-ab9e88adbf53-plugins-config-volume" pod "7ba87092-50fe-4937-9ddb-ab9e88adbf53" (UID: "7ba87092-50fe-4937-9ddb-ab9e88adbf53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:02.268268 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.268251    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdjnc\" (UniqueName: \"kubernetes.io/projected/f299fcdd-d631-4f4f-bb82-b32e5349ba76-kube-api-access-mdjnc\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:02.268325 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.268272    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f299fcdd-d631-4f4f-bb82-b32e5349ba76-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:02.268325 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.268282    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba87092-50fe-4937-9ddb-ab9e88adbf53-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:02.269816 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.269791    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba87092-50fe-4937-9ddb-ab9e88adbf53-kube-api-access-s78j6" pod "7ba87092-50fe-4937-9ddb-ab9e88adbf53" (UID: "7ba87092-50fe-4937-9ddb-ab9e88adbf53"). InnerVolumeSpecName "kube-api-access-s78j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:02.370336 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.370243    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s78j6\" (UniqueName: \"kubernetes.io/projected/7ba87092-50fe-4937-9ddb-ab9e88adbf53-kube-api-access-s78j6\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:02.665393 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.665308    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"]
Nov 02 09:53:02.666438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.666414    4449 cpu_manager.go:414] "containerMap: removing container" podUID="7ba87092-50fe-4937-9ddb-ab9e88adbf53" containerName="payload-pre-processing"
Nov 02 09:53:02.666501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.666443    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba87092-50fe-4937-9ddb-ab9e88adbf53" containerName="payload-pre-processing"
Nov 02 09:53:02.666501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.666476    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b4d02db6-2385-4eb2-98ce-fae2c070af0c" containerName="revoke-keys"
Nov 02 09:53:02.666501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.666485    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d02db6-2385-4eb2-98ce-fae2c070af0c" containerName="revoke-keys"
Nov 02 09:53:02.666679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.666519    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f299fcdd-d631-4f4f-bb82-b32e5349ba76" containerName="payload-processing"
Nov 02 09:53:02.666679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.666526    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f299fcdd-d631-4f4f-bb82-b32e5349ba76" containerName="payload-processing"
Nov 02 09:53:02.666806 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.666743    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d02db6-2385-4eb2-98ce-fae2c070af0c" containerName="revoke-keys"
Nov 02 09:53:02.666806 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.666762    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f299fcdd-d631-4f4f-bb82-b32e5349ba76" containerName="payload-processing"
Nov 02 09:53:02.666806 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.666783    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba87092-50fe-4937-9ddb-ab9e88adbf53" containerName="payload-pre-processing"
Nov 02 09:53:02.671044 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.671019    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.677978 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.677942    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-87e02313-openshift-default-dockercfg-xjbmp\""
Nov 02 09:53:02.678888 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.678862    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"]
Nov 02 09:53:02.794241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.791354    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-credential-socket\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.794241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.791421    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istiod-ca-cert\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.794241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.791472    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-data\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.794241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.791524    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-envoy\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.794241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.791580    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lclx\" (UniqueName: \"kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-kube-api-access-7lclx\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.794241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.791735    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-socket\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.794241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.791802    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-token\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.794241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.791851    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-certs\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.794241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.791917    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-podinfo\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.897239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.894560    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lclx\" (UniqueName: \"kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-kube-api-access-7lclx\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.897239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.894720    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-socket\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.897239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.894790    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-token\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.897239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.894834    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-certs\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.897239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.894894    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-podinfo\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.897239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.894937    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-credential-socket\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.897239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.894972    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istiod-ca-cert\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.897239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.895011    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-data\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.897239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.895057    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-envoy\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.897239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.897000    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-socket\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.898631 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.898603    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-envoy\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.899186 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.898998    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-certs\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.899326 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.899256    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-data\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.900330 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.900308    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istiod-ca-cert\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.900476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.898760    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-credential-socket\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.906089 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.906064    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-podinfo\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.913315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.913288    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lclx\" (UniqueName: \"kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-kube-api-access-7lclx\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.914281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.914259    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-token\") pod \"e2e-ait-87e02313-openshift-default-8749f6555-zhlms\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") " pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:02.968865 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.968777    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb" event={"ID":"7ba87092-50fe-4937-9ddb-ab9e88adbf53","Type":"ContainerDied","Data":"d71eec3a114550a1eb82c18ac467d81b5c31b27742fe4a416887ea6c55ce911a"}
Nov 02 09:53:02.968865 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.968829    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"
Nov 02 09:53:02.968865 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.968855    4449 scope.go:122] "RemoveContainer" containerID="03bc7af86fafc291aadeeb1627c5e45b45603e7c7cb1c67379af6610b10adca3"
Nov 02 09:53:02.970553 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.970510    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"
Nov 02 09:53:02.970653 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.970527    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz" event={"ID":"f299fcdd-d631-4f4f-bb82-b32e5349ba76","Type":"ContainerDied","Data":"69124f0dc78e96467679b952ac6d08c1fb52f2818ff6b6720b75599c5f34cbdc"}
Nov 02 09:53:02.985059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.985034    4449 scope.go:122] "RemoveContainer" containerID="64aaf8ad4f59063e3cf8f4eb805bd5a012347969bb9654a844eddd1f8b7c60c3"
Nov 02 09:53:02.991056 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:02.991029    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:03.007301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:03.005689    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"]
Nov 02 09:53:03.012287 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:03.012246    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-62e67514-74478d9975-7sdxb"]
Nov 02 09:53:03.027903 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:03.026356    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"]
Nov 02 09:53:03.035702 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:03.035670    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-62e67514-7b847995-hjrwz"]
Nov 02 09:53:03.209753 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:03.209722    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba87092-50fe-4937-9ddb-ab9e88adbf53" path="/var/lib/kubelet/pods/7ba87092-50fe-4937-9ddb-ab9e88adbf53/volumes"
Nov 02 09:53:03.210121 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:03.210108    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f299fcdd-d631-4f4f-bb82-b32e5349ba76" path="/var/lib/kubelet/pods/f299fcdd-d631-4f4f-bb82-b32e5349ba76/volumes"
Nov 02 09:53:03.322048 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:53:03.321980    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af7de5d_8b4e_43b3_ab61_82bdd06b62bf.slice/crio-f0e7fa4aa160d524bbed003f23f1322a6b3ebf273f313ecd0a56b3d2095cf868 WatchSource:0}: Error finding container f0e7fa4aa160d524bbed003f23f1322a6b3ebf273f313ecd0a56b3d2095cf868: Status 404 returned error can't find the container with id f0e7fa4aa160d524bbed003f23f1322a6b3ebf273f313ecd0a56b3d2095cf868
Nov 02 09:53:03.325965 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:03.325879    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:53:03.328673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:03.326747    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:53:03.328673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:03.326810    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:53:03.342679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:03.342652    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"]
Nov 02 09:53:03.982479 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:03.982439    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" event={"ID":"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf","Type":"ContainerStarted","Data":"6b86a61a4a97a56c52f38ac20c13fd888196eb90f5edf00619e04ec6dc80d363"}
Nov 02 09:53:03.982682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:03.982495    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" event={"ID":"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf","Type":"ContainerStarted","Data":"f0e7fa4aa160d524bbed003f23f1322a6b3ebf273f313ecd0a56b3d2095cf868"}
Nov 02 09:53:03.992010 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:03.991981    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:04.010001 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:04.009954    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" podStartSLOduration=2.009940262 podStartE2EDuration="2.009940262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:04.008357395 +0000 UTC m=+1060.946644170" watchObservedRunningTime="2026-11-02 09:53:04.009940262 +0000 UTC m=+1060.948227105"
Nov 02 09:53:05.057497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.051899    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"]
Nov 02 09:53:05.063552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.060474    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:05.063552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.060571    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:05.063552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.060597    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:05.063552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.060680    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.063552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.062362    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"]
Nov 02 09:53:05.174799 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.174764    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.175007 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.174924    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.175105 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.175086    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.175180 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.175164    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpf9\" (UniqueName: \"kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-kube-api-access-4vpf9\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.276077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.276042    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.276273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.276177    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.276325 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.276286    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpf9\" (UniqueName: \"kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-kube-api-access-4vpf9\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.276531 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.276501    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.276934 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.276915    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.277004 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.276983    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.279149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.279129    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.288611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.288588    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpf9\" (UniqueName: \"kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-kube-api-access-4vpf9\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.377915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.377826    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:05.708045 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:05.707993    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"]
Nov 02 09:53:05.710718 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:53:05.710691    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6692fe0c_a649_4c5f_aeb6_bd80777cd4c4.slice/crio-2c9413fb6031dc94789005a86fb8afbc4b453be3ea5e20ba03bf91419518ea3d WatchSource:0}: Error finding container 2c9413fb6031dc94789005a86fb8afbc4b453be3ea5e20ba03bf91419518ea3d: Status 404 returned error can't find the container with id 2c9413fb6031dc94789005a86fb8afbc4b453be3ea5e20ba03bf91419518ea3d
Nov 02 09:53:06.027924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.027831    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"]
Nov 02 09:53:06.033120 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.033092    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-069ae59f-68q25"]
Nov 02 09:53:06.068038 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.068002    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq" event={"ID":"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4","Type":"ContainerStarted","Data":"136623828a325cb459923109c3235396448a3a6bc7eecefab2422d63ec388814"}
Nov 02 09:53:06.068456 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.068061    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq" event={"ID":"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4","Type":"ContainerStarted","Data":"2c9413fb6031dc94789005a86fb8afbc4b453be3ea5e20ba03bf91419518ea3d"}
Nov 02 09:53:06.068456 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.068324    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:06.089545 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.089489    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq" podStartSLOduration=2.089474928 podStartE2EDuration="2.089474928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:06.086282216 +0000 UTC m=+1063.024569202" watchObservedRunningTime="2026-11-02 09:53:06.089474928 +0000 UTC m=+1063.027761703"
Nov 02 09:53:06.573812 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:53:06.573767    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:53:06.805762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.805727    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"]
Nov 02 09:53:06.810694 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.810673    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"
Nov 02 09:53:06.815611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.815585    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-87e02313-dockercfg-ltk24\""
Nov 02 09:53:06.815755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.815717    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-87e02313\""
Nov 02 09:53:06.822463 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.822441    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"]
Nov 02 09:53:06.851396 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.851309    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"]
Nov 02 09:53:06.870562 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.870529    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"
Nov 02 09:53:06.877416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.877385    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"]
Nov 02 09:53:06.976933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.976899    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4snvp\" (UniqueName: \"kubernetes.io/projected/3b327ec7-90a8-48dd-ae28-87ad85609e2e-kube-api-access-4snvp\") pod \"payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc\" (UID: \"3b327ec7-90a8-48dd-ae28-87ad85609e2e\") " pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"
Nov 02 09:53:06.978695 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.978653    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8qm\" (UniqueName: \"kubernetes.io/projected/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-kube-api-access-qn8qm\") pod \"payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst\" (UID: \"3ed58f2f-1e60-47cf-9fd4-299e62b51f95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"
Nov 02 09:53:06.978936 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.978919    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b327ec7-90a8-48dd-ae28-87ad85609e2e-plugins-config-volume\") pod \"payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc\" (UID: \"3b327ec7-90a8-48dd-ae28-87ad85609e2e\") " pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"
Nov 02 09:53:06.979201 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:06.979182    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst\" (UID: \"3ed58f2f-1e60-47cf-9fd4-299e62b51f95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"
Nov 02 09:53:07.080394 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.080359    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4snvp\" (UniqueName: \"kubernetes.io/projected/3b327ec7-90a8-48dd-ae28-87ad85609e2e-kube-api-access-4snvp\") pod \"payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc\" (UID: \"3b327ec7-90a8-48dd-ae28-87ad85609e2e\") " pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"
Nov 02 09:53:07.080907 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.080490    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8qm\" (UniqueName: \"kubernetes.io/projected/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-kube-api-access-qn8qm\") pod \"payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst\" (UID: \"3ed58f2f-1e60-47cf-9fd4-299e62b51f95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"
Nov 02 09:53:07.080907 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.080537    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b327ec7-90a8-48dd-ae28-87ad85609e2e-plugins-config-volume\") pod \"payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc\" (UID: \"3b327ec7-90a8-48dd-ae28-87ad85609e2e\") " pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"
Nov 02 09:53:07.080907 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.080665    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst\" (UID: \"3ed58f2f-1e60-47cf-9fd4-299e62b51f95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"
Nov 02 09:53:07.081384 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.081351    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b327ec7-90a8-48dd-ae28-87ad85609e2e-plugins-config-volume\") pod \"payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc\" (UID: \"3b327ec7-90a8-48dd-ae28-87ad85609e2e\") " pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"
Nov 02 09:53:07.081502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.081419    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst\" (UID: \"3ed58f2f-1e60-47cf-9fd4-299e62b51f95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"
Nov 02 09:53:07.099837 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.099806    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4snvp\" (UniqueName: \"kubernetes.io/projected/3b327ec7-90a8-48dd-ae28-87ad85609e2e-kube-api-access-4snvp\") pod \"payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc\" (UID: \"3b327ec7-90a8-48dd-ae28-87ad85609e2e\") " pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"
Nov 02 09:53:07.101586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.101515    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8qm\" (UniqueName: \"kubernetes.io/projected/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-kube-api-access-qn8qm\") pod \"payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst\" (UID: \"3ed58f2f-1e60-47cf-9fd4-299e62b51f95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"
Nov 02 09:53:07.130449 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.130412    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"
Nov 02 09:53:07.188284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.187884    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"
Nov 02 09:53:07.222318 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.221489    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e355051-bae8-4cfd-bf5a-ab816171cce1" path="/var/lib/kubelet/pods/9e355051-bae8-4cfd-bf5a-ab816171cce1/volumes"
Nov 02 09:53:07.222318 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.221994    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"]
Nov 02 09:53:07.234380 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.234353    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"
Nov 02 09:53:07.236897 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.236867    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"]
Nov 02 09:53:07.238543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.238506    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:53:07.238543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.238505    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:53:07.247490 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.247374    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:53:07.247490 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.247463    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:53:07.284350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.284319    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc0d550-dpmqz\" (UID: \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"
Nov 02 09:53:07.284578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.284560    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7w9t\" (UniqueName: \"kubernetes.io/projected/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-kube-api-access-c7w9t\") pod \"maas-api-revoke-sub-6cc0d550-dpmqz\" (UID: \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"
Nov 02 09:53:07.284870 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.284851    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc0d550-dpmqz\" (UID: \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"
Nov 02 09:53:07.386372 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.386268    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc0d550-dpmqz\" (UID: \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"
Nov 02 09:53:07.386562 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.386450    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc0d550-dpmqz\" (UID: \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"
Nov 02 09:53:07.386562 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.386482    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7w9t\" (UniqueName: \"kubernetes.io/projected/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-kube-api-access-c7w9t\") pod \"maas-api-revoke-sub-6cc0d550-dpmqz\" (UID: \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"
Nov 02 09:53:07.387048 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.387030    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc0d550-dpmqz\" (UID: \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"
Nov 02 09:53:07.388922 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.388889    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc0d550-dpmqz\" (UID: \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"
Nov 02 09:53:07.398843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.398821    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7w9t\" (UniqueName: \"kubernetes.io/projected/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-kube-api-access-c7w9t\") pod \"maas-api-revoke-sub-6cc0d550-dpmqz\" (UID: \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"
Nov 02 09:53:07.415909 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:53:07.415878    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b327ec7_90a8_48dd_ae28_87ad85609e2e.slice/crio-7eeaf544d822f19a9f6e130f9acf9fa3020e4eb2897b2b0522e4eff3388676e4 WatchSource:0}: Error finding container 7eeaf544d822f19a9f6e130f9acf9fa3020e4eb2897b2b0522e4eff3388676e4: Status 404 returned error can't find the container with id 7eeaf544d822f19a9f6e130f9acf9fa3020e4eb2897b2b0522e4eff3388676e4
Nov 02 09:53:07.417715 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.417662    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"]
Nov 02 09:53:07.506786 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:53:07.506756    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed58f2f_1e60_47cf_9fd4_299e62b51f95.slice/crio-3817db324dbfc718a1ea0a1853c80071bd54ff07788dd0df8b2c41ab3609d0cf WatchSource:0}: Error finding container 3817db324dbfc718a1ea0a1853c80071bd54ff07788dd0df8b2c41ab3609d0cf: Status 404 returned error can't find the container with id 3817db324dbfc718a1ea0a1853c80071bd54ff07788dd0df8b2c41ab3609d0cf
Nov 02 09:53:07.507865 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.507840    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"]
Nov 02 09:53:07.556702 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.556677    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"
Nov 02 09:53:07.850726 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:53:07.850696    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d9310b_570a_41d6_9eb3_54dd7aab79bf.slice/crio-abca12c0845d6b77119019eb4aa3f7e505c3414885d44734742bb9944337ceee WatchSource:0}: Error finding container abca12c0845d6b77119019eb4aa3f7e505c3414885d44734742bb9944337ceee: Status 404 returned error can't find the container with id abca12c0845d6b77119019eb4aa3f7e505c3414885d44734742bb9944337ceee
Nov 02 09:53:07.852539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:07.852518    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"]
Nov 02 09:53:08.089904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:08.089866    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc" event={"ID":"3b327ec7-90a8-48dd-ae28-87ad85609e2e","Type":"ContainerStarted","Data":"79ba5b42133ff52b215457f4c00cc55521ea8b0f4628f0df7979aae87250e9bc"}
Nov 02 09:53:08.090357 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:08.089925    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc" event={"ID":"3b327ec7-90a8-48dd-ae28-87ad85609e2e","Type":"ContainerStarted","Data":"7eeaf544d822f19a9f6e130f9acf9fa3020e4eb2897b2b0522e4eff3388676e4"}
Nov 02 09:53:08.090357 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:08.090003    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"
Nov 02 09:53:08.091482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:08.091459    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz" event={"ID":"d5d9310b-570a-41d6-9eb3-54dd7aab79bf","Type":"ContainerStarted","Data":"abca12c0845d6b77119019eb4aa3f7e505c3414885d44734742bb9944337ceee"}
Nov 02 09:53:08.093296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:08.093270    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst" event={"ID":"3ed58f2f-1e60-47cf-9fd4-299e62b51f95","Type":"ContainerStarted","Data":"05eb19bf76cf3e7e78f00b437ca284e0fcb103866d739932360b12ae87cb0298"}
Nov 02 09:53:08.093421 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:08.093323    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst" event={"ID":"3ed58f2f-1e60-47cf-9fd4-299e62b51f95","Type":"ContainerStarted","Data":"3817db324dbfc718a1ea0a1853c80071bd54ff07788dd0df8b2c41ab3609d0cf"}
Nov 02 09:53:08.093421 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:08.093352    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"
Nov 02 09:53:08.113862 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:08.113758    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc" podStartSLOduration=2.113743848 podStartE2EDuration="2.113743848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:08.107761032 +0000 UTC m=+1065.046047831" watchObservedRunningTime="2026-11-02 09:53:08.113743848 +0000 UTC m=+1065.052030663"
Nov 02 09:53:08.133053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:08.133002    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst" podStartSLOduration=2.132988556 podStartE2EDuration="2.132988556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:08.127264506 +0000 UTC m=+1065.065551304" watchObservedRunningTime="2026-11-02 09:53:08.132988556 +0000 UTC m=+1065.071275333"
Nov 02 09:53:09.104967 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:09.104930    4449 generic.go:356] "Generic (PLEG): container finished" podID="d5d9310b-570a-41d6-9eb3-54dd7aab79bf" containerID="1918b26dcf0c0d6e06a508874438723011103ef58b498f71fe803d51e935335e" exitCode=0
Nov 02 09:53:09.105481 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:09.105022    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz" event={"ID":"d5d9310b-570a-41d6-9eb3-54dd7aab79bf","Type":"ContainerDied","Data":"1918b26dcf0c0d6e06a508874438723011103ef58b498f71fe803d51e935335e"}
Nov 02 09:53:10.287843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:10.287816    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"
Nov 02 09:53:10.326816 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:10.326788    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-openshift-ca\") pod \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\" (UID: \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\") "
Nov 02 09:53:10.327015 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:10.326834    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-kube-api-access-c7w9t\" (UniqueName: \"kubernetes.io/projected/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-kube-api-access-c7w9t\") pod \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\" (UID: \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\") "
Nov 02 09:53:10.327015 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:10.326886    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-xks-ca\") pod \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\" (UID: \"d5d9310b-570a-41d6-9eb3-54dd7aab79bf\") "
Nov 02 09:53:10.327263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:10.327234    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-openshift-ca" pod "d5d9310b-570a-41d6-9eb3-54dd7aab79bf" (UID: "d5d9310b-570a-41d6-9eb3-54dd7aab79bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:10.327736 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:10.327709    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:10.329832 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:10.329791    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-xks-ca" pod "d5d9310b-570a-41d6-9eb3-54dd7aab79bf" (UID: "d5d9310b-570a-41d6-9eb3-54dd7aab79bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:10.330710 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:10.330675    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-kube-api-access-c7w9t" pod "d5d9310b-570a-41d6-9eb3-54dd7aab79bf" (UID: "d5d9310b-570a-41d6-9eb3-54dd7aab79bf"). InnerVolumeSpecName "kube-api-access-c7w9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:10.429495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:10.429414    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7w9t\" (UniqueName: \"kubernetes.io/projected/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-kube-api-access-c7w9t\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:10.429495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:10.429445    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d9310b-570a-41d6-9eb3-54dd7aab79bf-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:11.132416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:11.132376    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz" event={"ID":"d5d9310b-570a-41d6-9eb3-54dd7aab79bf","Type":"ContainerDied","Data":"abca12c0845d6b77119019eb4aa3f7e505c3414885d44734742bb9944337ceee"}
Nov 02 09:53:11.132708 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:11.132435    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abca12c0845d6b77119019eb4aa3f7e505c3414885d44734742bb9944337ceee"
Nov 02 09:53:11.132708 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:11.132393    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"
Nov 02 09:53:14.030392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.030353    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"]
Nov 02 09:53:14.040374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.039492    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3084fb1d-7qcnl"]
Nov 02 09:53:14.073337 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.073297    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"]
Nov 02 09:53:14.074145 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.074123    4449 cpu_manager.go:414] "containerMap: removing container" podUID="d5d9310b-570a-41d6-9eb3-54dd7aab79bf" containerName="revoke-keys"
Nov 02 09:53:14.074279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.074146    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d9310b-570a-41d6-9eb3-54dd7aab79bf" containerName="revoke-keys"
Nov 02 09:53:14.074336 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.074324    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d9310b-570a-41d6-9eb3-54dd7aab79bf" containerName="revoke-keys"
Nov 02 09:53:14.078514 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.078488    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"
Nov 02 09:53:14.083148 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.083114    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:53:14.083327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.083199    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:53:14.083423 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.083362    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:53:14.083480 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.083437    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:53:14.090663 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.090635    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"]
Nov 02 09:53:14.226944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.226899    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5740cd2-rwrj8\" (UID: \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"
Nov 02 09:53:14.232691 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.232655    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5740cd2-rwrj8\" (UID: \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"
Nov 02 09:53:14.233319 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.233205    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n72d8\" (UniqueName: \"kubernetes.io/projected/a7482ee0-db9f-4b9c-bb10-3eb75511d765-kube-api-access-n72d8\") pod \"maas-api-revoke-sub-e5740cd2-rwrj8\" (UID: \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"
Nov 02 09:53:14.337252 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.334777    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n72d8\" (UniqueName: \"kubernetes.io/projected/a7482ee0-db9f-4b9c-bb10-3eb75511d765-kube-api-access-n72d8\") pod \"maas-api-revoke-sub-e5740cd2-rwrj8\" (UID: \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"
Nov 02 09:53:14.337252 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.334937    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5740cd2-rwrj8\" (UID: \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"
Nov 02 09:53:14.337252 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.335030    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5740cd2-rwrj8\" (UID: \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"
Nov 02 09:53:14.337252 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.335874    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5740cd2-rwrj8\" (UID: \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"
Nov 02 09:53:14.357246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.355473    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5740cd2-rwrj8\" (UID: \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"
Nov 02 09:53:14.388250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.385927    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n72d8\" (UniqueName: \"kubernetes.io/projected/a7482ee0-db9f-4b9c-bb10-3eb75511d765-kube-api-access-n72d8\") pod \"maas-api-revoke-sub-e5740cd2-rwrj8\" (UID: \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"
Nov 02 09:53:14.416248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:14.414498    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"
Nov 02 09:53:15.233530 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:53:15.233334    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7482ee0_db9f_4b9c_bb10_3eb75511d765.slice/crio-40f3f4d55540884a152987696fef26bca13e8f61569696e365d894c3ac81b986 WatchSource:0}: Error finding container 40f3f4d55540884a152987696fef26bca13e8f61569696e365d894c3ac81b986: Status 404 returned error can't find the container with id 40f3f4d55540884a152987696fef26bca13e8f61569696e365d894c3ac81b986
Nov 02 09:53:15.234206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:15.234173    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5978d00-1798-4178-8ffd-7c3c76150f2f" path="/var/lib/kubelet/pods/f5978d00-1798-4178-8ffd-7c3c76150f2f/volumes"
Nov 02 09:53:15.234688 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:15.234669    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"]
Nov 02 09:53:15.326989 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:15.326957    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8" event={"ID":"a7482ee0-db9f-4b9c-bb10-3eb75511d765","Type":"ContainerStarted","Data":"40f3f4d55540884a152987696fef26bca13e8f61569696e365d894c3ac81b986"}
Nov 02 09:53:16.339546 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:16.339513    4449 generic.go:356] "Generic (PLEG): container finished" podID="a7482ee0-db9f-4b9c-bb10-3eb75511d765" containerID="a9459e0330b0aef63fba5ed2f75c8e2e6dd220ee5bc81ec51a26f54214ed5446" exitCode=0
Nov 02 09:53:16.339984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:16.339594    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8" event={"ID":"a7482ee0-db9f-4b9c-bb10-3eb75511d765","Type":"ContainerDied","Data":"a9459e0330b0aef63fba5ed2f75c8e2e6dd220ee5bc81ec51a26f54214ed5446"}
Nov 02 09:53:17.081495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:17.081461    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:53:17.142593 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:17.142555    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"]
Nov 02 09:53:17.529339 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:17.529309    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"
Nov 02 09:53:17.612802 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:17.612711    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-openshift-ca\") pod \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\" (UID: \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\") "
Nov 02 09:53:17.612982 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:17.612931    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7482ee0-db9f-4b9c-bb10-3eb75511d765-kube-api-access-n72d8\" (UniqueName: \"kubernetes.io/projected/a7482ee0-db9f-4b9c-bb10-3eb75511d765-kube-api-access-n72d8\") pod \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\" (UID: \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\") "
Nov 02 09:53:17.613076 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:17.613059    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-xks-ca\") pod \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\" (UID: \"a7482ee0-db9f-4b9c-bb10-3eb75511d765\") "
Nov 02 09:53:17.614627 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:17.614591    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-openshift-ca" pod "a7482ee0-db9f-4b9c-bb10-3eb75511d765" (UID: "a7482ee0-db9f-4b9c-bb10-3eb75511d765"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:17.617140 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:17.617066    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-xks-ca" pod "a7482ee0-db9f-4b9c-bb10-3eb75511d765" (UID: "a7482ee0-db9f-4b9c-bb10-3eb75511d765"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:17.623281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:17.621872    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7482ee0-db9f-4b9c-bb10-3eb75511d765-kube-api-access-n72d8" pod "a7482ee0-db9f-4b9c-bb10-3eb75511d765" (UID: "a7482ee0-db9f-4b9c-bb10-3eb75511d765"). InnerVolumeSpecName "kube-api-access-n72d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:17.714856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:17.714814    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n72d8\" (UniqueName: \"kubernetes.io/projected/a7482ee0-db9f-4b9c-bb10-3eb75511d765-kube-api-access-n72d8\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:17.714856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:17.714851    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:17.714856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:17.714861    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7482ee0-db9f-4b9c-bb10-3eb75511d765-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:18.028478 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:18.028382    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"]
Nov 02 09:53:18.034088 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:18.034058    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc6d3801-rccb5"]
Nov 02 09:53:18.362953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:18.362915    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"
Nov 02 09:53:18.363146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:18.362951    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8" event={"ID":"a7482ee0-db9f-4b9c-bb10-3eb75511d765","Type":"ContainerDied","Data":"40f3f4d55540884a152987696fef26bca13e8f61569696e365d894c3ac81b986"}
Nov 02 09:53:18.363146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:18.363001    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40f3f4d55540884a152987696fef26bca13e8f61569696e365d894c3ac81b986"
Nov 02 09:53:19.107893 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:19.107862    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"
Nov 02 09:53:19.108532 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:19.107926    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"
Nov 02 09:53:19.215802 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:19.215763    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e7ff987-a493-412b-a48c-c53c17e61823" path="/var/lib/kubelet/pods/3e7ff987-a493-412b-a48c-c53c17e61823/volumes"
Nov 02 09:53:20.113860 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.113824    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"]
Nov 02 09:53:20.114586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.114570    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a7482ee0-db9f-4b9c-bb10-3eb75511d765" containerName="revoke-keys"
Nov 02 09:53:20.114640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.114587    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7482ee0-db9f-4b9c-bb10-3eb75511d765" containerName="revoke-keys"
Nov 02 09:53:20.114735 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.114725    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7482ee0-db9f-4b9c-bb10-3eb75511d765" containerName="revoke-keys"
Nov 02 09:53:20.119057 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.119037    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"
Nov 02 09:53:20.123669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.123643    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:53:20.124135 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.124108    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:53:20.124307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.124184    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:53:20.124307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.124183    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:53:20.128102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.128078    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"]
Nov 02 09:53:20.253407 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.253371    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9\" (UID: \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"
Nov 02 09:53:20.253619 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.253498    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9\" (UID: \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"
Nov 02 09:53:20.253619 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.253597    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdtz\" (UniqueName: \"kubernetes.io/projected/28bda9cb-9bcf-40b5-81dc-67d423d936d1-kube-api-access-nxdtz\") pod \"maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9\" (UID: \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"
Nov 02 09:53:20.354929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.354891    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdtz\" (UniqueName: \"kubernetes.io/projected/28bda9cb-9bcf-40b5-81dc-67d423d936d1-kube-api-access-nxdtz\") pod \"maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9\" (UID: \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"
Nov 02 09:53:20.355204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.355181    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9\" (UID: \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"
Nov 02 09:53:20.355376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.355357    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9\" (UID: \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"
Nov 02 09:53:20.356066 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.356046    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9\" (UID: \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"
Nov 02 09:53:20.357880 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.357859    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9\" (UID: \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"
Nov 02 09:53:20.367444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.367395    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdtz\" (UniqueName: \"kubernetes.io/projected/28bda9cb-9bcf-40b5-81dc-67d423d936d1-kube-api-access-nxdtz\") pod \"maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9\" (UID: \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"
Nov 02 09:53:20.444757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.444719    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"
Nov 02 09:53:20.745423 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:20.745388    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"]
Nov 02 09:53:21.395624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:21.395533    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9" event={"ID":"28bda9cb-9bcf-40b5-81dc-67d423d936d1","Type":"ContainerStarted","Data":"f8384f4e75e8278bffb92a4d9484817dc9f631af634b9cf46b890562875f8fa0"}
Nov 02 09:53:21.395624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:21.395587    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9" event={"ID":"28bda9cb-9bcf-40b5-81dc-67d423d936d1","Type":"ContainerStarted","Data":"becfb52d43a32687ec28d59aaf6b61762a1edf5bd6341440bdf698cc2467786c"}
Nov 02 09:53:21.417414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:21.417355    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9" podStartSLOduration=1.417338057 podStartE2EDuration="1.417338057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:21.414523516 +0000 UTC m=+1078.352810321" watchObservedRunningTime="2026-11-02 09:53:21.417338057 +0000 UTC m=+1078.355624832"
Nov 02 09:53:22.143689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.143634    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb" podUID="b7872390-6d52-4e22-a7dd-0617bf443ec0" containerName="payload-processing" containerID="cri-o://3e96954ac4229fde7fade54ad17ac11b1e1cf95081145392e210979b533b2841" gracePeriod=25
Nov 02 09:53:22.414532 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.414438    4449 generic.go:356] "Generic (PLEG): container finished" podID="28bda9cb-9bcf-40b5-81dc-67d423d936d1" containerID="f8384f4e75e8278bffb92a4d9484817dc9f631af634b9cf46b890562875f8fa0" exitCode=0
Nov 02 09:53:22.437670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.414971    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9" event={"ID":"28bda9cb-9bcf-40b5-81dc-67d423d936d1","Type":"ContainerDied","Data":"f8384f4e75e8278bffb92a4d9484817dc9f631af634b9cf46b890562875f8fa0"}
Nov 02 09:53:22.453133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.453088    4449 generic.go:356] "Generic (PLEG): container finished" podID="b7872390-6d52-4e22-a7dd-0617bf443ec0" containerID="3e96954ac4229fde7fade54ad17ac11b1e1cf95081145392e210979b533b2841" exitCode=0
Nov 02 09:53:22.453349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.453277    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb" event={"ID":"b7872390-6d52-4e22-a7dd-0617bf443ec0","Type":"ContainerDied","Data":"3e96954ac4229fde7fade54ad17ac11b1e1cf95081145392e210979b533b2841"}
Nov 02 09:53:22.456376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.456352    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:53:22.594798 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.594762    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-provider-credentials\") pod \"b7872390-6d52-4e22-a7dd-0617bf443ec0\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") "
Nov 02 09:53:22.595369 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.594819    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-plugins-config-volume\") pod \"b7872390-6d52-4e22-a7dd-0617bf443ec0\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") "
Nov 02 09:53:22.595369 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.594940    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-kube-api-access-r9dlr\" (UniqueName: \"kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-kube-api-access-r9dlr\") pod \"b7872390-6d52-4e22-a7dd-0617bf443ec0\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") "
Nov 02 09:53:22.595369 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.595085    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-routing-overlay-volume\") pod \"b7872390-6d52-4e22-a7dd-0617bf443ec0\" (UID: \"b7872390-6d52-4e22-a7dd-0617bf443ec0\") "
Nov 02 09:53:22.596295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.596250    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-routing-overlay-volume" pod "b7872390-6d52-4e22-a7dd-0617bf443ec0" (UID: "b7872390-6d52-4e22-a7dd-0617bf443ec0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:22.598089 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.598051    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-plugins-config-volume" pod "b7872390-6d52-4e22-a7dd-0617bf443ec0" (UID: "b7872390-6d52-4e22-a7dd-0617bf443ec0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:22.599102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.599078    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-provider-credentials" pod "b7872390-6d52-4e22-a7dd-0617bf443ec0" (UID: "b7872390-6d52-4e22-a7dd-0617bf443ec0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:22.599868 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.599836    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-kube-api-access-r9dlr" pod "b7872390-6d52-4e22-a7dd-0617bf443ec0" (UID: "b7872390-6d52-4e22-a7dd-0617bf443ec0"). InnerVolumeSpecName "kube-api-access-r9dlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:22.697781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.697692    4449 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-provider-credentials\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:22.697781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.697725    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:22.697781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.697735    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9dlr\" (UniqueName: \"kubernetes.io/projected/b7872390-6d52-4e22-a7dd-0617bf443ec0-kube-api-access-r9dlr\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:22.697781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:22.697744    4449 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7872390-6d52-4e22-a7dd-0617bf443ec0-routing-overlay-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:23.464491 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.464460    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"
Nov 02 09:53:23.464937 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.464492    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb" event={"ID":"b7872390-6d52-4e22-a7dd-0617bf443ec0","Type":"ContainerDied","Data":"3770bc4819e7d6109af35f6f6622ee360cf235ce293f0f174534f97ef6ff51e9"}
Nov 02 09:53:23.464937 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.464564    4449 scope.go:122] "RemoveContainer" containerID="3e96954ac4229fde7fade54ad17ac11b1e1cf95081145392e210979b533b2841"
Nov 02 09:53:23.487108 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.487078    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"]
Nov 02 09:53:23.493291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.493253    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-9bfcq7gmb"]
Nov 02 09:53:23.641590 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.641551    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"
Nov 02 09:53:23.814427 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.814388    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28bda9cb-9bcf-40b5-81dc-67d423d936d1-kube-api-access-nxdtz\" (UniqueName: \"kubernetes.io/projected/28bda9cb-9bcf-40b5-81dc-67d423d936d1-kube-api-access-nxdtz\") pod \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\" (UID: \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\") "
Nov 02 09:53:23.814622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.814528    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-xks-ca\") pod \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\" (UID: \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\") "
Nov 02 09:53:23.814622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.814592    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-openshift-ca\") pod \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\" (UID: \"28bda9cb-9bcf-40b5-81dc-67d423d936d1\") "
Nov 02 09:53:23.815019 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.814978    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-openshift-ca" pod "28bda9cb-9bcf-40b5-81dc-67d423d936d1" (UID: "28bda9cb-9bcf-40b5-81dc-67d423d936d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:23.815210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.815186    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:23.817114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.817077    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-xks-ca" pod "28bda9cb-9bcf-40b5-81dc-67d423d936d1" (UID: "28bda9cb-9bcf-40b5-81dc-67d423d936d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:23.817114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.817089    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28bda9cb-9bcf-40b5-81dc-67d423d936d1-kube-api-access-nxdtz" pod "28bda9cb-9bcf-40b5-81dc-67d423d936d1" (UID: "28bda9cb-9bcf-40b5-81dc-67d423d936d1"). InnerVolumeSpecName "kube-api-access-nxdtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:23.868620 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.868583    4449 scope.go:122] "RemoveContainer" containerID="414f135912906b4615f7a3ba3705cda95f844fc4743912f3f3c273e759d0bb72"
Nov 02 09:53:23.881506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.881480    4449 scope.go:122] "RemoveContainer" containerID="57d2666344faedeb84653003d944885aafca19ec7ac5ddb0d86d7d609763e712"
Nov 02 09:53:23.893365 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.893343    4449 scope.go:122] "RemoveContainer" containerID="1a4e270bec975d6f28cfe23954c445a512a7559a271b40b066bebd9411b7b15c"
Nov 02 09:53:23.905159 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.905126    4449 scope.go:122] "RemoveContainer" containerID="d3079b62c3566433e4cd5571c379e8a366d262bfa7cf1c48930331db9f1366a1"
Nov 02 09:53:23.916591 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.916549    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxdtz\" (UniqueName: \"kubernetes.io/projected/28bda9cb-9bcf-40b5-81dc-67d423d936d1-kube-api-access-nxdtz\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:23.916758 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.916596    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28bda9cb-9bcf-40b5-81dc-67d423d936d1-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:23.919281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:23.918790    4449 scope.go:122] "RemoveContainer" containerID="dc2dc5f5c2b596787a2138089e39c739dd9d97c2bf2bc463a14629bb09b09774"
Nov 02 09:53:24.477045 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:24.477015    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"
Nov 02 09:53:24.477555 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:24.477038    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9" event={"ID":"28bda9cb-9bcf-40b5-81dc-67d423d936d1","Type":"ContainerDied","Data":"becfb52d43a32687ec28d59aaf6b61762a1edf5bd6341440bdf698cc2467786c"}
Nov 02 09:53:24.477555 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:24.477098    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="becfb52d43a32687ec28d59aaf6b61762a1edf5bd6341440bdf698cc2467786c"
Nov 02 09:53:25.209141 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.209109    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7872390-6d52-4e22-a7dd-0617bf443ec0" path="/var/lib/kubelet/pods/b7872390-6d52-4e22-a7dd-0617bf443ec0/volumes"
Nov 02 09:53:25.544733 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.544698    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"]
Nov 02 09:53:25.545488 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.545471    4449 cpu_manager.go:414] "containerMap: removing container" podUID="28bda9cb-9bcf-40b5-81dc-67d423d936d1" containerName="revoke-keys"
Nov 02 09:53:25.545548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.545493    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28bda9cb-9bcf-40b5-81dc-67d423d936d1" containerName="revoke-keys"
Nov 02 09:53:25.545548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.545511    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b7872390-6d52-4e22-a7dd-0617bf443ec0" containerName="payload-processing"
Nov 02 09:53:25.545548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.545518    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7872390-6d52-4e22-a7dd-0617bf443ec0" containerName="payload-processing"
Nov 02 09:53:25.545706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.545695    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="28bda9cb-9bcf-40b5-81dc-67d423d936d1" containerName="revoke-keys"
Nov 02 09:53:25.545740 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.545712    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7872390-6d52-4e22-a7dd-0617bf443ec0" containerName="payload-processing"
Nov 02 09:53:25.552002 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.551958    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"
Nov 02 09:53:25.557036 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.557000    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:53:25.557299 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.557272    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:53:25.557438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.557320    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:53:25.557438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.557278    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:53:25.557896 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.557876    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"]
Nov 02 09:53:25.642921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.642890    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc8714c0-5mnnl\" (UID: \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"
Nov 02 09:53:25.643111 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.642953    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx9j4\" (UniqueName: \"kubernetes.io/projected/0c968779-cd4c-4b68-9aa0-b2f1895a0558-kube-api-access-jx9j4\") pod \"maas-api-revoke-sub-cc8714c0-5mnnl\" (UID: \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"
Nov 02 09:53:25.643111 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.643098    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc8714c0-5mnnl\" (UID: \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"
Nov 02 09:53:25.743917 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.743889    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx9j4\" (UniqueName: \"kubernetes.io/projected/0c968779-cd4c-4b68-9aa0-b2f1895a0558-kube-api-access-jx9j4\") pod \"maas-api-revoke-sub-cc8714c0-5mnnl\" (UID: \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"
Nov 02 09:53:25.744089 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.743967    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc8714c0-5mnnl\" (UID: \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"
Nov 02 09:53:25.744129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.744118    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc8714c0-5mnnl\" (UID: \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"
Nov 02 09:53:25.744854 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.744761    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc8714c0-5mnnl\" (UID: \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"
Nov 02 09:53:25.746663 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.746639    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc8714c0-5mnnl\" (UID: \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"
Nov 02 09:53:25.758590 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.758569    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx9j4\" (UniqueName: \"kubernetes.io/projected/0c968779-cd4c-4b68-9aa0-b2f1895a0558-kube-api-access-jx9j4\") pod \"maas-api-revoke-sub-cc8714c0-5mnnl\" (UID: \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"
Nov 02 09:53:25.869656 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:25.869560    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"
Nov 02 09:53:26.306973 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:26.306924    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"]
Nov 02 09:53:26.497664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:26.497628    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl" event={"ID":"0c968779-cd4c-4b68-9aa0-b2f1895a0558","Type":"ContainerStarted","Data":"5e6c2aaad8df6d6396ed4c46abfdd20f4eb7d5f70bab8154142f9c39f1adfd75"}
Nov 02 09:53:27.031578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:27.031466    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"]
Nov 02 09:53:27.037599 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:27.037563    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35c15a1-mdcpw"]
Nov 02 09:53:27.229062 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:27.229024    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f25a86c-09f9-4304-9570-f13da4338ef3" path="/var/lib/kubelet/pods/5f25a86c-09f9-4304-9570-f13da4338ef3/volumes"
Nov 02 09:53:27.514429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:27.514387    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl" event={"ID":"0c968779-cd4c-4b68-9aa0-b2f1895a0558","Type":"ContainerStarted","Data":"8906a431f2c657b81583c2e745b2d529393b760b9a857dc5e137aab254ac4f84"}
Nov 02 09:53:27.541496 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:27.541434    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl" podStartSLOduration=2.541416112 podStartE2EDuration="2.541416112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:27.532434279 +0000 UTC m=+1084.470721073" watchObservedRunningTime="2026-11-02 09:53:27.541416112 +0000 UTC m=+1084.479702888"
Nov 02 09:53:28.558244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:28.557168    4449 generic.go:356] "Generic (PLEG): container finished" podID="0c968779-cd4c-4b68-9aa0-b2f1895a0558" containerID="8906a431f2c657b81583c2e745b2d529393b760b9a857dc5e137aab254ac4f84" exitCode=0
Nov 02 09:53:28.558244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:28.557432    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl" event={"ID":"0c968779-cd4c-4b68-9aa0-b2f1895a0558","Type":"ContainerDied","Data":"8906a431f2c657b81583c2e745b2d529393b760b9a857dc5e137aab254ac4f84"}
Nov 02 09:53:29.779170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:29.779141    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"
Nov 02 09:53:29.885249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:29.885128    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-openshift-ca\") pod \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\" (UID: \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\") "
Nov 02 09:53:29.885416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:29.885338    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c968779-cd4c-4b68-9aa0-b2f1895a0558-kube-api-access-jx9j4\" (UniqueName: \"kubernetes.io/projected/0c968779-cd4c-4b68-9aa0-b2f1895a0558-kube-api-access-jx9j4\") pod \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\" (UID: \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\") "
Nov 02 09:53:29.885416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:29.885388    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-xks-ca\") pod \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\" (UID: \"0c968779-cd4c-4b68-9aa0-b2f1895a0558\") "
Nov 02 09:53:29.885566 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:29.885545    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-openshift-ca" pod "0c968779-cd4c-4b68-9aa0-b2f1895a0558" (UID: "0c968779-cd4c-4b68-9aa0-b2f1895a0558"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:29.885980 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:29.885956    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:29.887726 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:29.887694    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c968779-cd4c-4b68-9aa0-b2f1895a0558-kube-api-access-jx9j4" pod "0c968779-cd4c-4b68-9aa0-b2f1895a0558" (UID: "0c968779-cd4c-4b68-9aa0-b2f1895a0558"). InnerVolumeSpecName "kube-api-access-jx9j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:29.888134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:29.888112    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-xks-ca" pod "0c968779-cd4c-4b68-9aa0-b2f1895a0558" (UID: "0c968779-cd4c-4b68-9aa0-b2f1895a0558"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:29.987157 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:29.987123    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx9j4\" (UniqueName: \"kubernetes.io/projected/0c968779-cd4c-4b68-9aa0-b2f1895a0558-kube-api-access-jx9j4\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:29.987157 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:29.987154    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c968779-cd4c-4b68-9aa0-b2f1895a0558-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:30.247686 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:30.247595    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"]
Nov 02 09:53:30.300542 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:30.300255    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"]
Nov 02 09:53:30.658240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:30.647572    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl" event={"ID":"0c968779-cd4c-4b68-9aa0-b2f1895a0558","Type":"ContainerDied","Data":"5e6c2aaad8df6d6396ed4c46abfdd20f4eb7d5f70bab8154142f9c39f1adfd75"}
Nov 02 09:53:30.658240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:30.647718    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e6c2aaad8df6d6396ed4c46abfdd20f4eb7d5f70bab8154142f9c39f1adfd75"
Nov 02 09:53:30.658240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:30.647802    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"
Nov 02 09:53:32.035532 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.035488    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"]
Nov 02 09:53:32.041262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.041234    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6e3572f-rrqpx"]
Nov 02 09:53:32.690039 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.690003    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"]
Nov 02 09:53:32.690770 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.690754    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0c968779-cd4c-4b68-9aa0-b2f1895a0558" containerName="revoke-keys"
Nov 02 09:53:32.690822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.690771    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c968779-cd4c-4b68-9aa0-b2f1895a0558" containerName="revoke-keys"
Nov 02 09:53:32.690909 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.690900    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c968779-cd4c-4b68-9aa0-b2f1895a0558" containerName="revoke-keys"
Nov 02 09:53:32.695150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.695115    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"
Nov 02 09:53:32.704808 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.704779    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"]
Nov 02 09:53:32.705051 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.705023    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:53:32.705166 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.705148    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:53:32.705411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.705390    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:53:32.705694 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.705680    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:53:32.843967 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.843934    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pvh9\" (UniqueName: \"kubernetes.io/projected/71d5d07c-71f9-455b-934a-05d4a2e7303c-kube-api-access-6pvh9\") pod \"maas-api-revoke-sub-87af7f2f-wcj92\" (UID: \"71d5d07c-71f9-455b-934a-05d4a2e7303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"
Nov 02 09:53:32.844152 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.843982    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87af7f2f-wcj92\" (UID: \"71d5d07c-71f9-455b-934a-05d4a2e7303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"
Nov 02 09:53:32.844152 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.844115    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87af7f2f-wcj92\" (UID: \"71d5d07c-71f9-455b-934a-05d4a2e7303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"
Nov 02 09:53:32.945802 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.945703    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pvh9\" (UniqueName: \"kubernetes.io/projected/71d5d07c-71f9-455b-934a-05d4a2e7303c-kube-api-access-6pvh9\") pod \"maas-api-revoke-sub-87af7f2f-wcj92\" (UID: \"71d5d07c-71f9-455b-934a-05d4a2e7303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"
Nov 02 09:53:32.946056 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.946032    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87af7f2f-wcj92\" (UID: \"71d5d07c-71f9-455b-934a-05d4a2e7303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"
Nov 02 09:53:32.946190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.946174    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87af7f2f-wcj92\" (UID: \"71d5d07c-71f9-455b-934a-05d4a2e7303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"
Nov 02 09:53:32.947622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.947583    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87af7f2f-wcj92\" (UID: \"71d5d07c-71f9-455b-934a-05d4a2e7303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"
Nov 02 09:53:32.950791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.950765    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87af7f2f-wcj92\" (UID: \"71d5d07c-71f9-455b-934a-05d4a2e7303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"
Nov 02 09:53:32.960176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:32.960146    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pvh9\" (UniqueName: \"kubernetes.io/projected/71d5d07c-71f9-455b-934a-05d4a2e7303c-kube-api-access-6pvh9\") pod \"maas-api-revoke-sub-87af7f2f-wcj92\" (UID: \"71d5d07c-71f9-455b-934a-05d4a2e7303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"
Nov 02 09:53:33.021492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:33.021455    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"
Nov 02 09:53:33.208763 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:33.208687    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac93e91-bcc2-4514-8f7b-15bbf0ca5286" path="/var/lib/kubelet/pods/9ac93e91-bcc2-4514-8f7b-15bbf0ca5286/volumes"
Nov 02 09:53:33.313476 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:53:33.313442    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d5d07c_71f9_455b_934a_05d4a2e7303c.slice/crio-e0032e4d3daf3c5d523d47dd13568a3658aebdf6965faf13cc755f2a49c72627 WatchSource:0}: Error finding container e0032e4d3daf3c5d523d47dd13568a3658aebdf6965faf13cc755f2a49c72627: Status 404 returned error can't find the container with id e0032e4d3daf3c5d523d47dd13568a3658aebdf6965faf13cc755f2a49c72627
Nov 02 09:53:33.314621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:33.314599    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"]
Nov 02 09:53:33.702817 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:33.702785    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92" event={"ID":"71d5d07c-71f9-455b-934a-05d4a2e7303c","Type":"ContainerStarted","Data":"9a8406f2ad95b729c5650ac0dc15d58456069dd213d5ab5a6b4e06dcc9fb34d8"}
Nov 02 09:53:33.702968 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:33.702833    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92" event={"ID":"71d5d07c-71f9-455b-934a-05d4a2e7303c","Type":"ContainerStarted","Data":"e0032e4d3daf3c5d523d47dd13568a3658aebdf6965faf13cc755f2a49c72627"}
Nov 02 09:53:33.725156 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:33.725102    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92" podStartSLOduration=1.725086399 podStartE2EDuration="1.725086399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:33.720697562 +0000 UTC m=+1090.658984337" watchObservedRunningTime="2026-11-02 09:53:33.725086399 +0000 UTC m=+1090.663373174"
Nov 02 09:53:34.716524 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:34.716427    4449 generic.go:356] "Generic (PLEG): container finished" podID="71d5d07c-71f9-455b-934a-05d4a2e7303c" containerID="9a8406f2ad95b729c5650ac0dc15d58456069dd213d5ab5a6b4e06dcc9fb34d8" exitCode=0
Nov 02 09:53:34.716952 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:34.716521    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92" event={"ID":"71d5d07c-71f9-455b-934a-05d4a2e7303c","Type":"ContainerDied","Data":"9a8406f2ad95b729c5650ac0dc15d58456069dd213d5ab5a6b4e06dcc9fb34d8"}
Nov 02 09:53:35.248508 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:35.248459    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb" podUID="b74a3871-b77b-41e3-85b8-f7bbb4e01651" containerName="payload-processing" containerID="cri-o://472bdace349c3ee8b68e7db7efbb255ef70435737d639af7e23f8e178a6de335" gracePeriod=25
Nov 02 09:53:35.301084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:35.301036    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6" podUID="0a77eafb-2709-47c6-870e-4fd9f7da6e6f" containerName="payload-pre-processing" containerID="cri-o://1ce9454695986e09fb6669e57518d6093747c05a39e93a4f0b799dd293e18e2d" gracePeriod=25
Nov 02 09:53:35.900652 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:35.900622    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"
Nov 02 09:53:35.991115 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:35.991008    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-xks-ca\") pod \"71d5d07c-71f9-455b-934a-05d4a2e7303c\" (UID: \"71d5d07c-71f9-455b-934a-05d4a2e7303c\") "
Nov 02 09:53:35.991334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:35.991154    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-openshift-ca\") pod \"71d5d07c-71f9-455b-934a-05d4a2e7303c\" (UID: \"71d5d07c-71f9-455b-934a-05d4a2e7303c\") "
Nov 02 09:53:35.991334 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:35.991312    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d5d07c-71f9-455b-934a-05d4a2e7303c-kube-api-access-6pvh9\" (UniqueName: \"kubernetes.io/projected/71d5d07c-71f9-455b-934a-05d4a2e7303c-kube-api-access-6pvh9\") pod \"71d5d07c-71f9-455b-934a-05d4a2e7303c\" (UID: \"71d5d07c-71f9-455b-934a-05d4a2e7303c\") "
Nov 02 09:53:35.992117 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:35.992041    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-openshift-ca" pod "71d5d07c-71f9-455b-934a-05d4a2e7303c" (UID: "71d5d07c-71f9-455b-934a-05d4a2e7303c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:35.994439 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:35.994401    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d5d07c-71f9-455b-934a-05d4a2e7303c-kube-api-access-6pvh9" pod "71d5d07c-71f9-455b-934a-05d4a2e7303c" (UID: "71d5d07c-71f9-455b-934a-05d4a2e7303c"). InnerVolumeSpecName "kube-api-access-6pvh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:35.995364 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:35.995333    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-xks-ca" pod "71d5d07c-71f9-455b-934a-05d4a2e7303c" (UID: "71d5d07c-71f9-455b-934a-05d4a2e7303c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:36.099015 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.098970    4449 patch_prober.go:29] interesting pod/e2e-ait-87e02313-openshift-default-8749f6555-zhlms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:53:36.099252 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.099143    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:53:36.099664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.099633    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:36.099784 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.099669    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d5d07c-71f9-455b-934a-05d4a2e7303c-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:36.099784 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.099687    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pvh9\" (UniqueName: \"kubernetes.io/projected/71d5d07c-71f9-455b-934a-05d4a2e7303c-kube-api-access-6pvh9\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:36.376456 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.376412    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"]
Nov 02 09:53:36.377148 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.377132    4449 cpu_manager.go:414] "containerMap: removing container" podUID="71d5d07c-71f9-455b-934a-05d4a2e7303c" containerName="revoke-keys"
Nov 02 09:53:36.377195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.377149    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d5d07c-71f9-455b-934a-05d4a2e7303c" containerName="revoke-keys"
Nov 02 09:53:36.377316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.377305    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d5d07c-71f9-455b-934a-05d4a2e7303c" containerName="revoke-keys"
Nov 02 09:53:36.381418 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.381400    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"
Nov 02 09:53:36.391186 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.391155    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"]
Nov 02 09:53:36.505943 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.505904    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dab2b34d-v485c\" (UID: \"ed6552c1-e93c-406e-8749-5796468c5aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"
Nov 02 09:53:36.506140 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.506034    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nfmr\" (UniqueName: \"kubernetes.io/projected/ed6552c1-e93c-406e-8749-5796468c5aba-kube-api-access-2nfmr\") pod \"maas-api-revoke-sub-dab2b34d-v485c\" (UID: \"ed6552c1-e93c-406e-8749-5796468c5aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"
Nov 02 09:53:36.506209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.506136    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dab2b34d-v485c\" (UID: \"ed6552c1-e93c-406e-8749-5796468c5aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"
Nov 02 09:53:36.549552 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:53:36.549485    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:53:36.607941 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.607898    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dab2b34d-v485c\" (UID: \"ed6552c1-e93c-406e-8749-5796468c5aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"
Nov 02 09:53:36.608110 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.607995    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nfmr\" (UniqueName: \"kubernetes.io/projected/ed6552c1-e93c-406e-8749-5796468c5aba-kube-api-access-2nfmr\") pod \"maas-api-revoke-sub-dab2b34d-v485c\" (UID: \"ed6552c1-e93c-406e-8749-5796468c5aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"
Nov 02 09:53:36.608110 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.608038    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dab2b34d-v485c\" (UID: \"ed6552c1-e93c-406e-8749-5796468c5aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"
Nov 02 09:53:36.608757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.608736    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dab2b34d-v485c\" (UID: \"ed6552c1-e93c-406e-8749-5796468c5aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"
Nov 02 09:53:36.610894 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.610867    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dab2b34d-v485c\" (UID: \"ed6552c1-e93c-406e-8749-5796468c5aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"
Nov 02 09:53:36.620523 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.620496    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nfmr\" (UniqueName: \"kubernetes.io/projected/ed6552c1-e93c-406e-8749-5796468c5aba-kube-api-access-2nfmr\") pod \"maas-api-revoke-sub-dab2b34d-v485c\" (UID: \"ed6552c1-e93c-406e-8749-5796468c5aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"
Nov 02 09:53:36.701634 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.701525    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"
Nov 02 09:53:36.736755 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.736724    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"
Nov 02 09:53:36.736890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.736750    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92" event={"ID":"71d5d07c-71f9-455b-934a-05d4a2e7303c","Type":"ContainerDied","Data":"e0032e4d3daf3c5d523d47dd13568a3658aebdf6965faf13cc755f2a49c72627"}
Nov 02 09:53:36.736890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.736801    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0032e4d3daf3c5d523d47dd13568a3658aebdf6965faf13cc755f2a49c72627"
Nov 02 09:53:36.922158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.922128    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"]
Nov 02 09:53:36.927760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.926550    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"
Nov 02 09:53:36.944928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.944895    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"]
Nov 02 09:53:36.989961 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:36.989928    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"]
Nov 02 09:53:36.994549 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:53:36.994515    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6552c1_e93c_406e_8749_5796468c5aba.slice/crio-74a4d8b83eb2ca6b379644b1f04ce93a458ce2d43b415590a966e3e5f79f1c02 WatchSource:0}: Error finding container 74a4d8b83eb2ca6b379644b1f04ce93a458ce2d43b415590a966e3e5f79f1c02: Status 404 returned error can't find the container with id 74a4d8b83eb2ca6b379644b1f04ce93a458ce2d43b415590a966e3e5f79f1c02
Nov 02 09:53:37.016392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.016364    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-87e02313-4rq7n\" (UID: \"d6d44db0-9559-48fc-960e-e192cceb67f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"
Nov 02 09:53:37.016502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.016408    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-87e02313-4rq7n\" (UID: \"d6d44db0-9559-48fc-960e-e192cceb67f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"
Nov 02 09:53:37.016502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.016474    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkf7t\" (UniqueName: \"kubernetes.io/projected/d6d44db0-9559-48fc-960e-e192cceb67f5-kube-api-access-rkf7t\") pod \"maas-api-revoke-keys-e2e-ait-87e02313-4rq7n\" (UID: \"d6d44db0-9559-48fc-960e-e192cceb67f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"
Nov 02 09:53:37.118298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.118257    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-87e02313-4rq7n\" (UID: \"d6d44db0-9559-48fc-960e-e192cceb67f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"
Nov 02 09:53:37.118491 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.118309    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-87e02313-4rq7n\" (UID: \"d6d44db0-9559-48fc-960e-e192cceb67f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"
Nov 02 09:53:37.118491 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.118373    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkf7t\" (UniqueName: \"kubernetes.io/projected/d6d44db0-9559-48fc-960e-e192cceb67f5-kube-api-access-rkf7t\") pod \"maas-api-revoke-keys-e2e-ait-87e02313-4rq7n\" (UID: \"d6d44db0-9559-48fc-960e-e192cceb67f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"
Nov 02 09:53:37.119079 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.119052    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-87e02313-4rq7n\" (UID: \"d6d44db0-9559-48fc-960e-e192cceb67f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"
Nov 02 09:53:37.120930 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.120908    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-87e02313-4rq7n\" (UID: \"d6d44db0-9559-48fc-960e-e192cceb67f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"
Nov 02 09:53:37.132942 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.132897    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkf7t\" (UniqueName: \"kubernetes.io/projected/d6d44db0-9559-48fc-960e-e192cceb67f5-kube-api-access-rkf7t\") pod \"maas-api-revoke-keys-e2e-ait-87e02313-4rq7n\" (UID: \"d6d44db0-9559-48fc-960e-e192cceb67f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"
Nov 02 09:53:37.333394 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.333358    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"
Nov 02 09:53:37.620509 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:53:37.620474    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d44db0_9559_48fc_960e_e192cceb67f5.slice/crio-24f5a900bc1d04626bace373afb3c6ae8bfa1325b20001d001d8eca02934ffd1 WatchSource:0}: Error finding container 24f5a900bc1d04626bace373afb3c6ae8bfa1325b20001d001d8eca02934ffd1: Status 404 returned error can't find the container with id 24f5a900bc1d04626bace373afb3c6ae8bfa1325b20001d001d8eca02934ffd1
Nov 02 09:53:37.622273 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.622249    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"]
Nov 02 09:53:37.761926 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.761894    4449 generic.go:356] "Generic (PLEG): container finished" podID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerID="6b86a61a4a97a56c52f38ac20c13fd888196eb90f5edf00619e04ec6dc80d363" exitCode=0
Nov 02 09:53:37.762132 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.761969    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" event={"ID":"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf","Type":"ContainerDied","Data":"6b86a61a4a97a56c52f38ac20c13fd888196eb90f5edf00619e04ec6dc80d363"}
Nov 02 09:53:37.762587 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.762564    4449 scope.go:122] "RemoveContainer" containerID="6b86a61a4a97a56c52f38ac20c13fd888196eb90f5edf00619e04ec6dc80d363"
Nov 02 09:53:37.764427 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.764393    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:53:37.764555 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.764414    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c" event={"ID":"ed6552c1-e93c-406e-8749-5796468c5aba","Type":"ContainerStarted","Data":"39e5aa0b819f2923c1dbf1910e39a1446dcaf1bd109cf2b24957c8d98052990c"}
Nov 02 09:53:37.764555 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.764501    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:53:37.764555 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.764536    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c" event={"ID":"ed6552c1-e93c-406e-8749-5796468c5aba","Type":"ContainerStarted","Data":"74a4d8b83eb2ca6b379644b1f04ce93a458ce2d43b415590a966e3e5f79f1c02"}
Nov 02 09:53:37.764555 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.764552    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:53:37.766139 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.766116    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n" event={"ID":"d6d44db0-9559-48fc-960e-e192cceb67f5","Type":"ContainerStarted","Data":"24f5a900bc1d04626bace373afb3c6ae8bfa1325b20001d001d8eca02934ffd1"}
Nov 02 09:53:37.854357 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:37.854249    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c" podStartSLOduration=1.85420878 podStartE2EDuration="1.85420878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:37.852620876 +0000 UTC m=+1094.790907663" watchObservedRunningTime="2026-11-02 09:53:37.85420878 +0000 UTC m=+1094.792495556"
Nov 02 09:53:38.793584 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:38.793554    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" event={"ID":"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf","Type":"ContainerStarted","Data":"ff33bf4e0ec73e3f22dd25bc764d76e0938e56617108d9f897594c642f9dcc13"}
Nov 02 09:53:38.795507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:38.795483    4449 generic.go:356] "Generic (PLEG): container finished" podID="ed6552c1-e93c-406e-8749-5796468c5aba" containerID="39e5aa0b819f2923c1dbf1910e39a1446dcaf1bd109cf2b24957c8d98052990c" exitCode=0
Nov 02 09:53:38.795632 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:38.795569    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c" event={"ID":"ed6552c1-e93c-406e-8749-5796468c5aba","Type":"ContainerDied","Data":"39e5aa0b819f2923c1dbf1910e39a1446dcaf1bd109cf2b24957c8d98052990c"}
Nov 02 09:53:38.797427 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:38.797400    4449 generic.go:356] "Generic (PLEG): container finished" podID="d6d44db0-9559-48fc-960e-e192cceb67f5" containerID="7492b6523876f2e7aa2fd1d0c6d80a0646c1dddab65e1e98fadb6d1bfd82d53e" exitCode=0
Nov 02 09:53:38.797533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:38.797460    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n" event={"ID":"d6d44db0-9559-48fc-960e-e192cceb67f5","Type":"ContainerDied","Data":"7492b6523876f2e7aa2fd1d0c6d80a0646c1dddab65e1e98fadb6d1bfd82d53e"}
Nov 02 09:53:38.991995 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:38.991957    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:38.992497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:38.992472    4449 patch_prober.go:29] interesting pod/e2e-ait-87e02313-openshift-default-8749f6555-zhlms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 02 09:53:38.992568 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:38.992523    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 02 09:53:39.994196 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:39.994159    4449 patch_prober.go:29] interesting pod/e2e-ait-87e02313-openshift-default-8749f6555-zhlms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 02 09:53:39.994710 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:39.994243    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 02 09:53:40.101030 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.101003    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"
Nov 02 09:53:40.105167 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.105145    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"
Nov 02 09:53:40.184872 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.184839    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-xks-ca\") pod \"d6d44db0-9559-48fc-960e-e192cceb67f5\" (UID: \"d6d44db0-9559-48fc-960e-e192cceb67f5\") "
Nov 02 09:53:40.185068 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.184964    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-openshift-ca\") pod \"d6d44db0-9559-48fc-960e-e192cceb67f5\" (UID: \"d6d44db0-9559-48fc-960e-e192cceb67f5\") "
Nov 02 09:53:40.185068 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.185003    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6552c1-e93c-406e-8749-5796468c5aba-kube-api-access-2nfmr\" (UniqueName: \"kubernetes.io/projected/ed6552c1-e93c-406e-8749-5796468c5aba-kube-api-access-2nfmr\") pod \"ed6552c1-e93c-406e-8749-5796468c5aba\" (UID: \"ed6552c1-e93c-406e-8749-5796468c5aba\") "
Nov 02 09:53:40.185182 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.185142    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-openshift-ca\") pod \"ed6552c1-e93c-406e-8749-5796468c5aba\" (UID: \"ed6552c1-e93c-406e-8749-5796468c5aba\") "
Nov 02 09:53:40.185182 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.185171    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-xks-ca\") pod \"ed6552c1-e93c-406e-8749-5796468c5aba\" (UID: \"ed6552c1-e93c-406e-8749-5796468c5aba\") "
Nov 02 09:53:40.185330 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.185201    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d44db0-9559-48fc-960e-e192cceb67f5-kube-api-access-rkf7t\" (UniqueName: \"kubernetes.io/projected/d6d44db0-9559-48fc-960e-e192cceb67f5-kube-api-access-rkf7t\") pod \"d6d44db0-9559-48fc-960e-e192cceb67f5\" (UID: \"d6d44db0-9559-48fc-960e-e192cceb67f5\") "
Nov 02 09:53:40.185838 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.185554    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-openshift-ca" pod "ed6552c1-e93c-406e-8749-5796468c5aba" (UID: "ed6552c1-e93c-406e-8749-5796468c5aba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:40.185997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.185973    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:40.186157 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.186137    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-openshift-ca" pod "d6d44db0-9559-48fc-960e-e192cceb67f5" (UID: "d6d44db0-9559-48fc-960e-e192cceb67f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:40.188085 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.188051    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-xks-ca" pod "ed6552c1-e93c-406e-8749-5796468c5aba" (UID: "ed6552c1-e93c-406e-8749-5796468c5aba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:40.188270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.188247    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d44db0-9559-48fc-960e-e192cceb67f5-kube-api-access-rkf7t" pod "d6d44db0-9559-48fc-960e-e192cceb67f5" (UID: "d6d44db0-9559-48fc-960e-e192cceb67f5"). InnerVolumeSpecName "kube-api-access-rkf7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:40.188342 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.188287    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6552c1-e93c-406e-8749-5796468c5aba-kube-api-access-2nfmr" pod "ed6552c1-e93c-406e-8749-5796468c5aba" (UID: "ed6552c1-e93c-406e-8749-5796468c5aba"). InnerVolumeSpecName "kube-api-access-2nfmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:40.188390 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.188379    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-xks-ca" pod "d6d44db0-9559-48fc-960e-e192cceb67f5" (UID: "d6d44db0-9559-48fc-960e-e192cceb67f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:40.287681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.287651    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nfmr\" (UniqueName: \"kubernetes.io/projected/ed6552c1-e93c-406e-8749-5796468c5aba-kube-api-access-2nfmr\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:40.287681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.287681    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6552c1-e93c-406e-8749-5796468c5aba-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:40.287681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.287696    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkf7t\" (UniqueName: \"kubernetes.io/projected/d6d44db0-9559-48fc-960e-e192cceb67f5-kube-api-access-rkf7t\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:40.287921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.287710    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:40.287921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.287723    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d44db0-9559-48fc-960e-e192cceb67f5-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:40.818933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.818907    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"
Nov 02 09:53:40.819113 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.818940    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c" event={"ID":"ed6552c1-e93c-406e-8749-5796468c5aba","Type":"ContainerDied","Data":"74a4d8b83eb2ca6b379644b1f04ce93a458ce2d43b415590a966e3e5f79f1c02"}
Nov 02 09:53:40.819113 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.819004    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74a4d8b83eb2ca6b379644b1f04ce93a458ce2d43b415590a966e3e5f79f1c02"
Nov 02 09:53:40.820583 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.820565    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"
Nov 02 09:53:40.820718 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.820560    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n" event={"ID":"d6d44db0-9559-48fc-960e-e192cceb67f5","Type":"ContainerDied","Data":"24f5a900bc1d04626bace373afb3c6ae8bfa1325b20001d001d8eca02934ffd1"}
Nov 02 09:53:40.820718 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.820694    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24f5a900bc1d04626bace373afb3c6ae8bfa1325b20001d001d8eca02934ffd1"
Nov 02 09:53:40.992088 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.992055    4449 patch_prober.go:29] interesting pod/e2e-ait-87e02313-openshift-default-8749f6555-zhlms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 02 09:53:40.992321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:40.992112    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 02 09:53:41.393243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:41.393172    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6" podUID="0a77eafb-2709-47c6-870e-4fd9f7da6e6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:53:41.393243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:41.393197    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb" podUID="b74a3871-b77b-41e3-85b8-f7bbb4e01651" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:53:41.991546 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:41.991510    4449 patch_prober.go:29] interesting pod/e2e-ait-87e02313-openshift-default-8749f6555-zhlms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 02 09:53:41.991735 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:41.991568    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 02 09:53:42.029762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:42.029728    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"]
Nov 02 09:53:42.035473 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:42.035442    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-553494eb-xlm2n"]
Nov 02 09:53:42.992026 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:42.991991    4449 patch_prober.go:29] interesting pod/e2e-ait-87e02313-openshift-default-8749f6555-zhlms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 02 09:53:42.992448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:42.992069    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 02 09:53:43.209210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:43.209172    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e01f7d9d-391b-4f4a-9cd0-31bff22bfd88" path="/var/lib/kubelet/pods/e01f7d9d-391b-4f4a-9cd0-31bff22bfd88/volumes"
Nov 02 09:53:43.991474 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:43.991433    4449 patch_prober.go:29] interesting pod/e2e-ait-87e02313-openshift-default-8749f6555-zhlms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 02 09:53:43.991657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:43.991499    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 02 09:53:44.998929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:44.998896    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:44.999497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:44.999478    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:45.000448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:45.000422    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:47.803093 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:47.803049    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"]
Nov 02 09:53:47.856301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:47.856266    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"]
Nov 02 09:53:48.259036 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:48.257014    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.156:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:53:48.334763 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:48.334285    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n" podUID="7968e57e-6e72-4ef2-af9f-82c111d870f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:53:50.507632 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.507590    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"]
Nov 02 09:53:50.508774 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.508751    4449 cpu_manager.go:414] "containerMap: removing container" podUID="d6d44db0-9559-48fc-960e-e192cceb67f5" containerName="revoke-keys"
Nov 02 09:53:50.508860 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.508778    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d44db0-9559-48fc-960e-e192cceb67f5" containerName="revoke-keys"
Nov 02 09:53:50.508860 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.508818    4449 cpu_manager.go:414] "containerMap: removing container" podUID="ed6552c1-e93c-406e-8749-5796468c5aba" containerName="revoke-keys"
Nov 02 09:53:50.508860 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.508826    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6552c1-e93c-406e-8749-5796468c5aba" containerName="revoke-keys"
Nov 02 09:53:50.509062 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.509042    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6552c1-e93c-406e-8749-5796468c5aba" containerName="revoke-keys"
Nov 02 09:53:50.509207 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.509077    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d44db0-9559-48fc-960e-e192cceb67f5" containerName="revoke-keys"
Nov 02 09:53:50.517728 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.517700    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"
Nov 02 09:53:50.520879 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.520851    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"]
Nov 02 09:53:50.522635 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.522614    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:53:50.522776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.522660    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:53:50.522776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.522743    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:53:50.522922 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.522846    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:53:50.673139 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.673107    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e30f837a-cx7pl\" (UID: \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"
Nov 02 09:53:50.673320 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.673257    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e30f837a-cx7pl\" (UID: \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"
Nov 02 09:53:50.673377 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.673352    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcd6b\" (UniqueName: \"kubernetes.io/projected/336e15ff-516c-4baa-bd54-82ddfd4c25cd-kube-api-access-kcd6b\") pod \"maas-api-revoke-sub-e30f837a-cx7pl\" (UID: \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"
Nov 02 09:53:50.779946 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.779812    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e30f837a-cx7pl\" (UID: \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"
Nov 02 09:53:50.780311 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.780279    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e30f837a-cx7pl\" (UID: \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"
Nov 02 09:53:50.780450 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.780378    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcd6b\" (UniqueName: \"kubernetes.io/projected/336e15ff-516c-4baa-bd54-82ddfd4c25cd-kube-api-access-kcd6b\") pod \"maas-api-revoke-sub-e30f837a-cx7pl\" (UID: \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"
Nov 02 09:53:50.782170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.782144    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e30f837a-cx7pl\" (UID: \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"
Nov 02 09:53:50.787298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.787272    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e30f837a-cx7pl\" (UID: \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"
Nov 02 09:53:50.793112 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.793081    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcd6b\" (UniqueName: \"kubernetes.io/projected/336e15ff-516c-4baa-bd54-82ddfd4c25cd-kube-api-access-kcd6b\") pod \"maas-api-revoke-sub-e30f837a-cx7pl\" (UID: \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"
Nov 02 09:53:50.837025 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:50.836992    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"
Nov 02 09:53:51.127516 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:53:51.127488    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336e15ff_516c_4baa_bd54_82ddfd4c25cd.slice/crio-44ea1e9c0e55ba71744ede40e772a5f3e77bec98b81d3b765fb19cad8158a03e WatchSource:0}: Error finding container 44ea1e9c0e55ba71744ede40e772a5f3e77bec98b81d3b765fb19cad8158a03e: Status 404 returned error can't find the container with id 44ea1e9c0e55ba71744ede40e772a5f3e77bec98b81d3b765fb19cad8158a03e
Nov 02 09:53:51.131670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:51.131646    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"]
Nov 02 09:53:51.323240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:51.322885    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl" event={"ID":"336e15ff-516c-4baa-bd54-82ddfd4c25cd","Type":"ContainerStarted","Data":"44ea1e9c0e55ba71744ede40e772a5f3e77bec98b81d3b765fb19cad8158a03e"}
Nov 02 09:53:51.394195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:51.394102    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb" podUID="b74a3871-b77b-41e3-85b8-f7bbb4e01651" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:53:51.394368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:51.394109    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6" podUID="0a77eafb-2709-47c6-870e-4fd9f7da6e6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:53:51.431282 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:51.431244    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"]
Nov 02 09:53:51.439718 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:51.439689    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e02313-4rq7n"]
Nov 02 09:53:52.069486 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:52.069450    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"]
Nov 02 09:53:52.069986 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:52.069717    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy" containerID="cri-o://ff33bf4e0ec73e3f22dd25bc764d76e0938e56617108d9f897594c642f9dcc13" gracePeriod=30
Nov 02 09:53:52.338907 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:52.338800    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl" event={"ID":"336e15ff-516c-4baa-bd54-82ddfd4c25cd","Type":"ContainerStarted","Data":"772a22865c7b0d95f94899ec8031b351cfc72121367b1039160afcf991d94536"}
Nov 02 09:53:52.363040 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:52.362994    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl" podStartSLOduration=2.362977927 podStartE2EDuration="2.362977927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:52.360119453 +0000 UTC m=+1109.298406229" watchObservedRunningTime="2026-11-02 09:53:52.362977927 +0000 UTC m=+1109.301264702"
Nov 02 09:53:52.803984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:52.803920    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc" podUID="3b327ec7-90a8-48dd-ae28-87ad85609e2e" containerName="payload-processing" containerID="cri-o://79ba5b42133ff52b215457f4c00cc55521ea8b0f4628f0df7979aae87250e9bc" gracePeriod=25
Nov 02 09:53:52.857060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:52.857003    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst" podUID="3ed58f2f-1e60-47cf-9fd4-299e62b51f95" containerName="payload-pre-processing" containerID="cri-o://05eb19bf76cf3e7e78f00b437ca284e0fcb103866d739932360b12ae87cb0298" gracePeriod=25
Nov 02 09:53:53.157351 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.157318    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"]
Nov 02 09:53:53.164120 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.163639    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"
Nov 02 09:53:53.172177 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.167031    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.172867 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.172843    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-13c6a8db-openshift-default-dockercfg-2kqnp\""
Nov 02 09:53:53.186050 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.186024    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"]
Nov 02 09:53:53.216875 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.216839    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d44db0-9559-48fc-960e-e192cceb67f5" path="/var/lib/kubelet/pods/d6d44db0-9559-48fc-960e-e192cceb67f5/volumes"
Nov 02 09:53:53.239116 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.239084    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"
Nov 02 09:53:53.251460 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.251430    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b327ec7-90a8-48dd-ae28-87ad85609e2e-kube-api-access-4snvp\" (UniqueName: \"kubernetes.io/projected/3b327ec7-90a8-48dd-ae28-87ad85609e2e-kube-api-access-4snvp\") pod \"3b327ec7-90a8-48dd-ae28-87ad85609e2e\" (UID: \"3b327ec7-90a8-48dd-ae28-87ad85609e2e\") "
Nov 02 09:53:53.251667 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.251497    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b327ec7-90a8-48dd-ae28-87ad85609e2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b327ec7-90a8-48dd-ae28-87ad85609e2e-plugins-config-volume\") pod \"3b327ec7-90a8-48dd-ae28-87ad85609e2e\" (UID: \"3b327ec7-90a8-48dd-ae28-87ad85609e2e\") "
Nov 02 09:53:53.251667 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.251596    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-kube-api-access-qn8qm\" (UniqueName: \"kubernetes.io/projected/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-kube-api-access-qn8qm\") pod \"3ed58f2f-1e60-47cf-9fd4-299e62b51f95\" (UID: \"3ed58f2f-1e60-47cf-9fd4-299e62b51f95\") "
Nov 02 09:53:53.252047 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.251675    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-plugins-config-volume\") pod \"3ed58f2f-1e60-47cf-9fd4-299e62b51f95\" (UID: \"3ed58f2f-1e60-47cf-9fd4-299e62b51f95\") "
Nov 02 09:53:53.252047 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.251881    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1934300-75bb-44de-b982-25d0fd418c46-istiod-ca-cert\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.252047 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.252034    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-envoy\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.252209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.252036    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b327ec7-90a8-48dd-ae28-87ad85609e2e-plugins-config-volume" pod "3b327ec7-90a8-48dd-ae28-87ad85609e2e" (UID: "3b327ec7-90a8-48dd-ae28-87ad85609e2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:53.252209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.252120    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2mdl\" (UniqueName: \"kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-kube-api-access-v2mdl\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.252209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.252200    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-certs\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.252452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.252297    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-istio-token\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.252452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.252366    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-socket\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.252717 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.252695    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-plugins-config-volume" pod "3ed58f2f-1e60-47cf-9fd4-299e62b51f95" (UID: "3ed58f2f-1e60-47cf-9fd4-299e62b51f95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:53.252814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.252750    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-credential-socket\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.252912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.252885    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1934300-75bb-44de-b982-25d0fd418c46-istio-podinfo\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.253033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.253015    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-data\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.253154 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.253141    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b327ec7-90a8-48dd-ae28-87ad85609e2e-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:53.253203 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.253162    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:53.253997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.253977    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b327ec7-90a8-48dd-ae28-87ad85609e2e-kube-api-access-4snvp" pod "3b327ec7-90a8-48dd-ae28-87ad85609e2e" (UID: "3b327ec7-90a8-48dd-ae28-87ad85609e2e"). InnerVolumeSpecName "kube-api-access-4snvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:53.254092 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.254074    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-kube-api-access-qn8qm" pod "3ed58f2f-1e60-47cf-9fd4-299e62b51f95" (UID: "3ed58f2f-1e60-47cf-9fd4-299e62b51f95"). InnerVolumeSpecName "kube-api-access-qn8qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.380123    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-envoy\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.380193    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mdl\" (UniqueName: \"kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-kube-api-access-v2mdl\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.380273    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-certs\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.380295    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-istio-token\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.380331    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-socket\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.380406    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-credential-socket\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.380493    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1934300-75bb-44de-b982-25d0fd418c46-istio-podinfo\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.380598    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-data\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.380669    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1934300-75bb-44de-b982-25d0fd418c46-istiod-ca-cert\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.380766    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4snvp\" (UniqueName: \"kubernetes.io/projected/3b327ec7-90a8-48dd-ae28-87ad85609e2e-kube-api-access-4snvp\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.380783    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn8qm\" (UniqueName: \"kubernetes.io/projected/3ed58f2f-1e60-47cf-9fd4-299e62b51f95-kube-api-access-qn8qm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.381669    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1934300-75bb-44de-b982-25d0fd418c46-istiod-ca-cert\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.384721    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-socket\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.390292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.384985    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-credential-socket\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.402267 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.391795    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-data\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.402267 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.392420    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-certs\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.405530 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.405476    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-envoy\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.406255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.406190    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1934300-75bb-44de-b982-25d0fd418c46-istio-podinfo\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.482242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.464029    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2mdl\" (UniqueName: \"kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-kube-api-access-v2mdl\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.487258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.486036    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-istio-token\") pod \"e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") " pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.490657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.490611    4449 generic.go:356] "Generic (PLEG): container finished" podID="3ed58f2f-1e60-47cf-9fd4-299e62b51f95" containerID="05eb19bf76cf3e7e78f00b437ca284e0fcb103866d739932360b12ae87cb0298" exitCode=0
Nov 02 09:53:53.490793 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.490715    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst" event={"ID":"3ed58f2f-1e60-47cf-9fd4-299e62b51f95","Type":"ContainerDied","Data":"05eb19bf76cf3e7e78f00b437ca284e0fcb103866d739932360b12ae87cb0298"}
Nov 02 09:53:53.490890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.490833    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"
Nov 02 09:53:53.490890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.490834    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst" event={"ID":"3ed58f2f-1e60-47cf-9fd4-299e62b51f95","Type":"ContainerDied","Data":"3817db324dbfc718a1ea0a1853c80071bd54ff07788dd0df8b2c41ab3609d0cf"}
Nov 02 09:53:53.491011 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.490892    4449 scope.go:122] "RemoveContainer" containerID="05eb19bf76cf3e7e78f00b437ca284e0fcb103866d739932360b12ae87cb0298"
Nov 02 09:53:53.492886 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.492850    4449 generic.go:356] "Generic (PLEG): container finished" podID="3b327ec7-90a8-48dd-ae28-87ad85609e2e" containerID="79ba5b42133ff52b215457f4c00cc55521ea8b0f4628f0df7979aae87250e9bc" exitCode=0
Nov 02 09:53:53.493025 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.492931    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"
Nov 02 09:53:53.493092 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.492931    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc" event={"ID":"3b327ec7-90a8-48dd-ae28-87ad85609e2e","Type":"ContainerDied","Data":"79ba5b42133ff52b215457f4c00cc55521ea8b0f4628f0df7979aae87250e9bc"}
Nov 02 09:53:53.493092 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.493075    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc" event={"ID":"3b327ec7-90a8-48dd-ae28-87ad85609e2e","Type":"ContainerDied","Data":"7eeaf544d822f19a9f6e130f9acf9fa3020e4eb2897b2b0522e4eff3388676e4"}
Nov 02 09:53:53.495955 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.495930    4449 generic.go:356] "Generic (PLEG): container finished" podID="336e15ff-516c-4baa-bd54-82ddfd4c25cd" containerID="772a22865c7b0d95f94899ec8031b351cfc72121367b1039160afcf991d94536" exitCode=0
Nov 02 09:53:53.496098 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.496085    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl" event={"ID":"336e15ff-516c-4baa-bd54-82ddfd4c25cd","Type":"ContainerDied","Data":"772a22865c7b0d95f94899ec8031b351cfc72121367b1039160afcf991d94536"}
Nov 02 09:53:53.506251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.506233    4449 scope.go:122] "RemoveContainer" containerID="05eb19bf76cf3e7e78f00b437ca284e0fcb103866d739932360b12ae87cb0298"
Nov 02 09:53:53.506521 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:53:53.506500    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05eb19bf76cf3e7e78f00b437ca284e0fcb103866d739932360b12ae87cb0298\": container with ID starting with 05eb19bf76cf3e7e78f00b437ca284e0fcb103866d739932360b12ae87cb0298 not found: ID does not exist" containerID="05eb19bf76cf3e7e78f00b437ca284e0fcb103866d739932360b12ae87cb0298"
Nov 02 09:53:53.506610 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.506532    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05eb19bf76cf3e7e78f00b437ca284e0fcb103866d739932360b12ae87cb0298"} err="failed to get container status \"05eb19bf76cf3e7e78f00b437ca284e0fcb103866d739932360b12ae87cb0298\": rpc error: code = NotFound desc = could not find container \"05eb19bf76cf3e7e78f00b437ca284e0fcb103866d739932360b12ae87cb0298\": container with ID starting with 05eb19bf76cf3e7e78f00b437ca284e0fcb103866d739932360b12ae87cb0298 not found: ID does not exist"
Nov 02 09:53:53.506610 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.506554    4449 scope.go:122] "RemoveContainer" containerID="79ba5b42133ff52b215457f4c00cc55521ea8b0f4628f0df7979aae87250e9bc"
Nov 02 09:53:53.518935 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.518908    4449 scope.go:122] "RemoveContainer" containerID="79ba5b42133ff52b215457f4c00cc55521ea8b0f4628f0df7979aae87250e9bc"
Nov 02 09:53:53.519312 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:53:53.519285    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ba5b42133ff52b215457f4c00cc55521ea8b0f4628f0df7979aae87250e9bc\": container with ID starting with 79ba5b42133ff52b215457f4c00cc55521ea8b0f4628f0df7979aae87250e9bc not found: ID does not exist" containerID="79ba5b42133ff52b215457f4c00cc55521ea8b0f4628f0df7979aae87250e9bc"
Nov 02 09:53:53.519849 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.519807    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ba5b42133ff52b215457f4c00cc55521ea8b0f4628f0df7979aae87250e9bc"} err="failed to get container status \"79ba5b42133ff52b215457f4c00cc55521ea8b0f4628f0df7979aae87250e9bc\": rpc error: code = NotFound desc = could not find container \"79ba5b42133ff52b215457f4c00cc55521ea8b0f4628f0df7979aae87250e9bc\": container with ID starting with 79ba5b42133ff52b215457f4c00cc55521ea8b0f4628f0df7979aae87250e9bc not found: ID does not exist"
Nov 02 09:53:53.539543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.539515    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:53.552152 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.552119    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"]
Nov 02 09:53:53.560486 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.560449    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87e02313-b88c5bc4f-4v9cc"]
Nov 02 09:53:53.578375 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.578347    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"]
Nov 02 09:53:53.591394 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.591366    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87e02313-5df95b66cf-lqlst"]
Nov 02 09:53:53.939667 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.939483    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"]
Nov 02 09:53:53.945328 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.945279    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:53:53.945534 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.945512    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:53:53.945654 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:53.945637    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:53:54.115466 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.115382    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"]
Nov 02 09:53:54.122448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.122415    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9cbd331c-k5fv9"]
Nov 02 09:53:54.486471 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.486399    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"]
Nov 02 09:53:54.487092 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.487075    4449 cpu_manager.go:414] "containerMap: removing container" podUID="3ed58f2f-1e60-47cf-9fd4-299e62b51f95" containerName="payload-pre-processing"
Nov 02 09:53:54.487092 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.487091    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed58f2f-1e60-47cf-9fd4-299e62b51f95" containerName="payload-pre-processing"
Nov 02 09:53:54.487197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.487106    4449 cpu_manager.go:414] "containerMap: removing container" podUID="3b327ec7-90a8-48dd-ae28-87ad85609e2e" containerName="payload-processing"
Nov 02 09:53:54.487197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.487111    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b327ec7-90a8-48dd-ae28-87ad85609e2e" containerName="payload-processing"
Nov 02 09:53:54.487287 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.487278    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed58f2f-1e60-47cf-9fd4-299e62b51f95" containerName="payload-pre-processing"
Nov 02 09:53:54.487321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.487288    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b327ec7-90a8-48dd-ae28-87ad85609e2e" containerName="payload-processing"
Nov 02 09:53:54.491411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.491391    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"
Nov 02 09:53:54.512612 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.510837    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"]
Nov 02 09:53:54.523207 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.523172    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" event={"ID":"e1934300-75bb-44de-b982-25d0fd418c46","Type":"ContainerStarted","Data":"fa9782e034b757ed2bbeba83032cea1677860b9c47103939f822b8f660fa9fbf"}
Nov 02 09:53:54.523393 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.523328    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" event={"ID":"e1934300-75bb-44de-b982-25d0fd418c46","Type":"ContainerStarted","Data":"5641c8300e9214f729016bfcf3bc77ba130648a271b22de005d902ab9f5d0921"}
Nov 02 09:53:54.540160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.540130    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:54.558279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.553999    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" podStartSLOduration=1.5539772859999998 podStartE2EDuration="1.553977286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:54.544885779 +0000 UTC m=+1111.483172555" watchObservedRunningTime="2026-11-02 09:53:54.553977286 +0000 UTC m=+1111.492264063"
Nov 02 09:53:54.571248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.568597    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ced1b7\""
Nov 02 09:53:54.571248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.568747    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ced1b7-dockercfg-vhvlk\""
Nov 02 09:53:54.607085 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.607040    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qknx5\" (UniqueName: \"kubernetes.io/projected/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-kube-api-access-qknx5\") pod \"payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k\" (UID: \"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99\") " pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"
Nov 02 09:53:54.607303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.607272    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-plugins-config-volume\") pod \"payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k\" (UID: \"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99\") " pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"
Nov 02 09:53:54.716692 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.712622    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qknx5\" (UniqueName: \"kubernetes.io/projected/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-kube-api-access-qknx5\") pod \"payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k\" (UID: \"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99\") " pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"
Nov 02 09:53:54.716692 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.712784    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-plugins-config-volume\") pod \"payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k\" (UID: \"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99\") " pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"
Nov 02 09:53:54.716692 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.713869    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-plugins-config-volume\") pod \"payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k\" (UID: \"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99\") " pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"
Nov 02 09:53:54.750280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.749777    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qknx5\" (UniqueName: \"kubernetes.io/projected/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-kube-api-access-qknx5\") pod \"payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k\" (UID: \"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99\") " pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"
Nov 02 09:53:54.817906 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.817877    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"
Nov 02 09:53:54.883319 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.883291    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"
Nov 02 09:53:54.916953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.916920    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-xks-ca\") pod \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\" (UID: \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\") "
Nov 02 09:53:54.917103 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.917008    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-openshift-ca\") pod \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\" (UID: \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\") "
Nov 02 09:53:54.917319 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.917282    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336e15ff-516c-4baa-bd54-82ddfd4c25cd-kube-api-access-kcd6b\" (UniqueName: \"kubernetes.io/projected/336e15ff-516c-4baa-bd54-82ddfd4c25cd-kube-api-access-kcd6b\") pod \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\" (UID: \"336e15ff-516c-4baa-bd54-82ddfd4c25cd\") "
Nov 02 09:53:54.918243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.918194    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-openshift-ca" pod "336e15ff-516c-4baa-bd54-82ddfd4c25cd" (UID: "336e15ff-516c-4baa-bd54-82ddfd4c25cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:54.920822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.920796    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336e15ff-516c-4baa-bd54-82ddfd4c25cd-kube-api-access-kcd6b" pod "336e15ff-516c-4baa-bd54-82ddfd4c25cd" (UID: "336e15ff-516c-4baa-bd54-82ddfd4c25cd"). InnerVolumeSpecName "kube-api-access-kcd6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:54.921545 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:54.921516    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-xks-ca" pod "336e15ff-516c-4baa-bd54-82ddfd4c25cd" (UID: "336e15ff-516c-4baa-bd54-82ddfd4c25cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:55.020022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.019937    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcd6b\" (UniqueName: \"kubernetes.io/projected/336e15ff-516c-4baa-bd54-82ddfd4c25cd-kube-api-access-kcd6b\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:55.020022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.019971    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:55.020022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.019981    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/336e15ff-516c-4baa-bd54-82ddfd4c25cd-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:55.025261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.025200    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"]
Nov 02 09:53:55.030534 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.030509    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f4b7b2-2gvvc"]
Nov 02 09:53:55.120054 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:53:55.120021    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a2e7a6_b8f8_45d3_8d54_614f7ba17b99.slice/crio-143c8272d86d8f6ae91db08519817282489110adcb1321873d78948305c84355 WatchSource:0}: Error finding container 143c8272d86d8f6ae91db08519817282489110adcb1321873d78948305c84355: Status 404 returned error can't find the container with id 143c8272d86d8f6ae91db08519817282489110adcb1321873d78948305c84355
Nov 02 09:53:55.121585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.121557    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"]
Nov 02 09:53:55.230564 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.230539    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28bda9cb-9bcf-40b5-81dc-67d423d936d1" path="/var/lib/kubelet/pods/28bda9cb-9bcf-40b5-81dc-67d423d936d1/volumes"
Nov 02 09:53:55.231011 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.230996    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b327ec7-90a8-48dd-ae28-87ad85609e2e" path="/var/lib/kubelet/pods/3b327ec7-90a8-48dd-ae28-87ad85609e2e/volumes"
Nov 02 09:53:55.231446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.231431    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed58f2f-1e60-47cf-9fd4-299e62b51f95" path="/var/lib/kubelet/pods/3ed58f2f-1e60-47cf-9fd4-299e62b51f95/volumes"
Nov 02 09:53:55.232189 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.232167    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f895e8-16a7-4246-9722-c6955d275a44" path="/var/lib/kubelet/pods/b1f895e8-16a7-4246-9722-c6955d275a44/volumes"
Nov 02 09:53:55.538803 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.538775    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"
Nov 02 09:53:55.538803 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.538812    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl" event={"ID":"336e15ff-516c-4baa-bd54-82ddfd4c25cd","Type":"ContainerDied","Data":"44ea1e9c0e55ba71744ede40e772a5f3e77bec98b81d3b765fb19cad8158a03e"}
Nov 02 09:53:55.539318 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.538884    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44ea1e9c0e55ba71744ede40e772a5f3e77bec98b81d3b765fb19cad8158a03e"
Nov 02 09:53:55.542250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.542181    4449 generic.go:356] "Generic (PLEG): container finished" podID="b74a3871-b77b-41e3-85b8-f7bbb4e01651" containerID="472bdace349c3ee8b68e7db7efbb255ef70435737d639af7e23f8e178a6de335" exitCode=0
Nov 02 09:53:55.542479 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.542282    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb" event={"ID":"b74a3871-b77b-41e3-85b8-f7bbb4e01651","Type":"ContainerDied","Data":"472bdace349c3ee8b68e7db7efbb255ef70435737d639af7e23f8e178a6de335"}
Nov 02 09:53:55.545285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.545059    4449 generic.go:356] "Generic (PLEG): container finished" podID="0a77eafb-2709-47c6-870e-4fd9f7da6e6f" containerID="1ce9454695986e09fb6669e57518d6093747c05a39e93a4f0b799dd293e18e2d" exitCode=0
Nov 02 09:53:55.545285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.545197    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6" event={"ID":"0a77eafb-2709-47c6-870e-4fd9f7da6e6f","Type":"ContainerDied","Data":"1ce9454695986e09fb6669e57518d6093747c05a39e93a4f0b799dd293e18e2d"}
Nov 02 09:53:55.546621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.546596    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:55.549890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.549374    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k" event={"ID":"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99","Type":"ContainerStarted","Data":"a526d6b9e1dcf59ead21fe8a8da6a316bac547c34fb7016018282324d706539b"}
Nov 02 09:53:55.549890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.549420    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k" event={"ID":"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99","Type":"ContainerStarted","Data":"143c8272d86d8f6ae91db08519817282489110adcb1321873d78948305c84355"}
Nov 02 09:53:55.549890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.549813    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"
Nov 02 09:53:55.549890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.549869    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:55.550634 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.550517    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:53:55.562803 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.562742    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"
Nov 02 09:53:55.590559 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.590516    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k" podStartSLOduration=1.590502507 podStartE2EDuration="1.590502507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:55.586245548 +0000 UTC m=+1112.524532364" watchObservedRunningTime="2026-11-02 09:53:55.590502507 +0000 UTC m=+1112.528789298"
Nov 02 09:53:55.627941 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.627915    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"
Nov 02 09:53:55.628173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.628153    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74a3871-b77b-41e3-85b8-f7bbb4e01651-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74a3871-b77b-41e3-85b8-f7bbb4e01651-plugins-config-volume\") pod \"b74a3871-b77b-41e3-85b8-f7bbb4e01651\" (UID: \"b74a3871-b77b-41e3-85b8-f7bbb4e01651\") "
Nov 02 09:53:55.628266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.628249    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74a3871-b77b-41e3-85b8-f7bbb4e01651-kube-api-access-lnnnj\" (UniqueName: \"kubernetes.io/projected/b74a3871-b77b-41e3-85b8-f7bbb4e01651-kube-api-access-lnnnj\") pod \"b74a3871-b77b-41e3-85b8-f7bbb4e01651\" (UID: \"b74a3871-b77b-41e3-85b8-f7bbb4e01651\") "
Nov 02 09:53:55.628613 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.628585    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74a3871-b77b-41e3-85b8-f7bbb4e01651-plugins-config-volume" pod "b74a3871-b77b-41e3-85b8-f7bbb4e01651" (UID: "b74a3871-b77b-41e3-85b8-f7bbb4e01651"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:55.629266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.629204    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b74a3871-b77b-41e3-85b8-f7bbb4e01651-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:55.643269 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.631329    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74a3871-b77b-41e3-85b8-f7bbb4e01651-kube-api-access-lnnnj" pod "b74a3871-b77b-41e3-85b8-f7bbb4e01651" (UID: "b74a3871-b77b-41e3-85b8-f7bbb4e01651"). InnerVolumeSpecName "kube-api-access-lnnnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:55.734511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.734466    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-kube-api-access-dhp45\" (UniqueName: \"kubernetes.io/projected/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-kube-api-access-dhp45\") pod \"0a77eafb-2709-47c6-870e-4fd9f7da6e6f\" (UID: \"0a77eafb-2709-47c6-870e-4fd9f7da6e6f\") "
Nov 02 09:53:55.736897 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.736475    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-plugins-config-volume\") pod \"0a77eafb-2709-47c6-870e-4fd9f7da6e6f\" (UID: \"0a77eafb-2709-47c6-870e-4fd9f7da6e6f\") "
Nov 02 09:53:55.747015 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.736850    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-plugins-config-volume" pod "0a77eafb-2709-47c6-870e-4fd9f7da6e6f" (UID: "0a77eafb-2709-47c6-870e-4fd9f7da6e6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:55.747301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.738869    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-kube-api-access-dhp45" pod "0a77eafb-2709-47c6-870e-4fd9f7da6e6f" (UID: "0a77eafb-2709-47c6-870e-4fd9f7da6e6f"). InnerVolumeSpecName "kube-api-access-dhp45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:55.748033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.748009    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:55.748135 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.748036    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnnnj\" (UniqueName: \"kubernetes.io/projected/b74a3871-b77b-41e3-85b8-f7bbb4e01651-kube-api-access-lnnnj\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:55.748135 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:55.748049    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhp45\" (UniqueName: \"kubernetes.io/projected/0a77eafb-2709-47c6-870e-4fd9f7da6e6f-kube-api-access-dhp45\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:56.562893 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:56.562863    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"
Nov 02 09:53:56.562893 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:56.562869    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb" event={"ID":"b74a3871-b77b-41e3-85b8-f7bbb4e01651","Type":"ContainerDied","Data":"526f3d59e8c54ec44eadf49293334c40d218a8bf84f4127cbe7ed91c10f133f9"}
Nov 02 09:53:56.563459 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:56.562937    4449 scope.go:122] "RemoveContainer" containerID="472bdace349c3ee8b68e7db7efbb255ef70435737d639af7e23f8e178a6de335"
Nov 02 09:53:56.564506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:56.564490    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"
Nov 02 09:53:56.564590 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:56.564529    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6" event={"ID":"0a77eafb-2709-47c6-870e-4fd9f7da6e6f","Type":"ContainerDied","Data":"c259e3fa03c6476a1c42495f0c7ba42392cb75566e04e9b535982393f49f3401"}
Nov 02 09:53:56.575830 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:56.575816    4449 scope.go:122] "RemoveContainer" containerID="1ce9454695986e09fb6669e57518d6093747c05a39e93a4f0b799dd293e18e2d"
Nov 02 09:53:56.593574 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:56.593547    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"]
Nov 02 09:53:56.602166 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:56.602146    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-qscjb"]
Nov 02 09:53:56.614151 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:56.614129    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"]
Nov 02 09:53:56.619043 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:56.619024    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-4jmx6"]
Nov 02 09:53:57.211858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.211824    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a77eafb-2709-47c6-870e-4fd9f7da6e6f" path="/var/lib/kubelet/pods/0a77eafb-2709-47c6-870e-4fd9f7da6e6f/volumes"
Nov 02 09:53:57.212481 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.212448    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74a3871-b77b-41e3-85b8-f7bbb4e01651" path="/var/lib/kubelet/pods/b74a3871-b77b-41e3-85b8-f7bbb4e01651/volumes"
Nov 02 09:53:57.358347 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.358318    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:57.480115 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.480082    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-socket\") pod \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") "
Nov 02 09:53:57.480441 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.480143    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-envoy\") pod \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") "
Nov 02 09:53:57.480441 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.480240    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-kube-api-access-7lclx\" (UniqueName: \"kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-kube-api-access-7lclx\") pod \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") "
Nov 02 09:53:57.480441 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.480270    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istiod-ca-cert\") pod \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") "
Nov 02 09:53:57.480441 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.480329    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-data\") pod \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") "
Nov 02 09:53:57.480441 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.480361    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-certs\") pod \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") "
Nov 02 09:53:57.480441 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.480407    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-credential-socket\") pod \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") "
Nov 02 09:53:57.480441 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.480439    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-token\" (UniqueName: \"kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-token\") pod \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") "
Nov 02 09:53:57.480805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.480475    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-socket" pod "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" (UID: "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:53:57.480805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.480608    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-podinfo\") pod \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\" (UID: \"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf\") "
Nov 02 09:53:57.480805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.480754    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istiod-ca-cert" pod "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" (UID: "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:57.481345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.481275    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-data" pod "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" (UID: "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:53:57.481345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.481284    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:57.481345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.481293    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-credential-socket" pod "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" (UID: "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:53:57.481345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.481305    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:57.481874 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.481691    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-certs" pod "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" (UID: "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:53:57.484059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.484030    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-envoy" pod "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" (UID: "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:53:57.485339 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.485308    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-kube-api-access-7lclx" pod "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" (UID: "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf"). InnerVolumeSpecName "kube-api-access-7lclx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:57.485448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.485334    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-podinfo" pod "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" (UID: "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:53:57.485448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.485411    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-token" pod "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" (UID: "6af7de5d-8b4e-43b3-ab61-82bdd06b62bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:57.557764 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.557729    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"]
Nov 02 09:53:57.558406 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558392    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b74a3871-b77b-41e3-85b8-f7bbb4e01651" containerName="payload-processing"
Nov 02 09:53:57.558475 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558408    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74a3871-b77b-41e3-85b8-f7bbb4e01651" containerName="payload-processing"
Nov 02 09:53:57.558475 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558445    4449 cpu_manager.go:414] "containerMap: removing container" podUID="336e15ff-516c-4baa-bd54-82ddfd4c25cd" containerName="revoke-keys"
Nov 02 09:53:57.558475 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558450    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336e15ff-516c-4baa-bd54-82ddfd4c25cd" containerName="revoke-keys"
Nov 02 09:53:57.558475 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558469    4449 cpu_manager.go:414] "containerMap: removing container" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy"
Nov 02 09:53:57.558475 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558474    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy"
Nov 02 09:53:57.558640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558485    4449 cpu_manager.go:414] "containerMap: removing container" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy"
Nov 02 09:53:57.558640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558489    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy"
Nov 02 09:53:57.558640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558497    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0a77eafb-2709-47c6-870e-4fd9f7da6e6f" containerName="payload-pre-processing"
Nov 02 09:53:57.558640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558503    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a77eafb-2709-47c6-870e-4fd9f7da6e6f" containerName="payload-pre-processing"
Nov 02 09:53:57.558640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558587    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74a3871-b77b-41e3-85b8-f7bbb4e01651" containerName="payload-processing"
Nov 02 09:53:57.558640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558600    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="336e15ff-516c-4baa-bd54-82ddfd4c25cd" containerName="revoke-keys"
Nov 02 09:53:57.558640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558610    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy"
Nov 02 09:53:57.558640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.558616    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a77eafb-2709-47c6-870e-4fd9f7da6e6f" containerName="payload-pre-processing"
Nov 02 09:53:57.562715 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.562695    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"
Nov 02 09:53:57.571024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.570995    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"]
Nov 02 09:53:57.571469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.571454    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-13c6a8db\""
Nov 02 09:53:57.572813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.572607    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-13c6a8db-dockercfg-fvjkq\""
Nov 02 09:53:57.582338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.582273    4449 generic.go:356] "Generic (PLEG): container finished" podID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerID="ff33bf4e0ec73e3f22dd25bc764d76e0938e56617108d9f897594c642f9dcc13" exitCode=0
Nov 02 09:53:57.582455 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.582349    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" event={"ID":"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf","Type":"ContainerDied","Data":"ff33bf4e0ec73e3f22dd25bc764d76e0938e56617108d9f897594c642f9dcc13"}
Nov 02 09:53:57.582455 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.582386    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms" event={"ID":"6af7de5d-8b4e-43b3-ab61-82bdd06b62bf","Type":"ContainerDied","Data":"f0e7fa4aa160d524bbed003f23f1322a6b3ebf273f313ecd0a56b3d2095cf868"}
Nov 02 09:53:57.582455 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.582410    4449 scope.go:122] "RemoveContainer" containerID="ff33bf4e0ec73e3f22dd25bc764d76e0938e56617108d9f897594c642f9dcc13"
Nov 02 09:53:57.582622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.582494    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"
Nov 02 09:53:57.582622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.582505    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:57.582622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.582532    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:57.582622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.582547    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:57.582622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.582562    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lclx\" (UniqueName: \"kubernetes.io/projected/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-kube-api-access-7lclx\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:57.582622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.582577    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:57.582861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.582688    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:57.582938 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.582915    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:57.622495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.622467    4449 scope.go:122] "RemoveContainer" containerID="6b86a61a4a97a56c52f38ac20c13fd888196eb90f5edf00619e04ec6dc80d363"
Nov 02 09:53:57.630249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.630210    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"]
Nov 02 09:53:57.631166 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.631141    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" containerName="istio-proxy"
Nov 02 09:53:57.636762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.636740    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"]
Nov 02 09:53:57.636867 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.636846    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"
Nov 02 09:53:57.679262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.671192    4449 scope.go:122] "RemoveContainer" containerID="ff33bf4e0ec73e3f22dd25bc764d76e0938e56617108d9f897594c642f9dcc13"
Nov 02 09:53:57.722193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.695336    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e13bb8-dd69-470b-b395-769e72deb289-plugins-config-volume\") pod \"payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2\" (UID: \"b4e13bb8-dd69-470b-b395-769e72deb289\") " pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"
Nov 02 09:53:57.722193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.695395    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae067e1-9dd1-4445-bd69-4c19f184559d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl\" (UID: \"5ae067e1-9dd1-4445-bd69-4c19f184559d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"
Nov 02 09:53:57.722193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.695515    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5s2v\" (UniqueName: \"kubernetes.io/projected/b4e13bb8-dd69-470b-b395-769e72deb289-kube-api-access-b5s2v\") pod \"payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2\" (UID: \"b4e13bb8-dd69-470b-b395-769e72deb289\") " pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"
Nov 02 09:53:57.722193 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.695701    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xlp4\" (UniqueName: \"kubernetes.io/projected/5ae067e1-9dd1-4445-bd69-4c19f184559d-kube-api-access-8xlp4\") pod \"payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl\" (UID: \"5ae067e1-9dd1-4445-bd69-4c19f184559d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"
Nov 02 09:53:57.729479 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:53:57.729430    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff33bf4e0ec73e3f22dd25bc764d76e0938e56617108d9f897594c642f9dcc13\": container with ID starting with ff33bf4e0ec73e3f22dd25bc764d76e0938e56617108d9f897594c642f9dcc13 not found: ID does not exist" containerID="ff33bf4e0ec73e3f22dd25bc764d76e0938e56617108d9f897594c642f9dcc13"
Nov 02 09:53:57.729673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.729493    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff33bf4e0ec73e3f22dd25bc764d76e0938e56617108d9f897594c642f9dcc13"} err="failed to get container status \"ff33bf4e0ec73e3f22dd25bc764d76e0938e56617108d9f897594c642f9dcc13\": rpc error: code = NotFound desc = could not find container \"ff33bf4e0ec73e3f22dd25bc764d76e0938e56617108d9f897594c642f9dcc13\": container with ID starting with ff33bf4e0ec73e3f22dd25bc764d76e0938e56617108d9f897594c642f9dcc13 not found: ID does not exist"
Nov 02 09:53:57.729673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.729525    4449 scope.go:122] "RemoveContainer" containerID="6b86a61a4a97a56c52f38ac20c13fd888196eb90f5edf00619e04ec6dc80d363"
Nov 02 09:53:57.773245 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:53:57.751919    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b86a61a4a97a56c52f38ac20c13fd888196eb90f5edf00619e04ec6dc80d363\": container with ID starting with 6b86a61a4a97a56c52f38ac20c13fd888196eb90f5edf00619e04ec6dc80d363 not found: ID does not exist" containerID="6b86a61a4a97a56c52f38ac20c13fd888196eb90f5edf00619e04ec6dc80d363"
Nov 02 09:53:57.773245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.751971    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b86a61a4a97a56c52f38ac20c13fd888196eb90f5edf00619e04ec6dc80d363"} err="failed to get container status \"6b86a61a4a97a56c52f38ac20c13fd888196eb90f5edf00619e04ec6dc80d363\": rpc error: code = NotFound desc = could not find container \"6b86a61a4a97a56c52f38ac20c13fd888196eb90f5edf00619e04ec6dc80d363\": container with ID starting with 6b86a61a4a97a56c52f38ac20c13fd888196eb90f5edf00619e04ec6dc80d363 not found: ID does not exist"
Nov 02 09:53:57.799268 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.788318    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"]
Nov 02 09:53:57.799268 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.794988    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-87e02313-openshift-default-8749f6555-zhlms"]
Nov 02 09:53:57.799268 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.797206    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e13bb8-dd69-470b-b395-769e72deb289-plugins-config-volume\") pod \"payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2\" (UID: \"b4e13bb8-dd69-470b-b395-769e72deb289\") " pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"
Nov 02 09:53:57.799268 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.797278    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae067e1-9dd1-4445-bd69-4c19f184559d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl\" (UID: \"5ae067e1-9dd1-4445-bd69-4c19f184559d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"
Nov 02 09:53:57.799268 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.797433    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5s2v\" (UniqueName: \"kubernetes.io/projected/b4e13bb8-dd69-470b-b395-769e72deb289-kube-api-access-b5s2v\") pod \"payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2\" (UID: \"b4e13bb8-dd69-470b-b395-769e72deb289\") " pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"
Nov 02 09:53:57.799268 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.797589    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xlp4\" (UniqueName: \"kubernetes.io/projected/5ae067e1-9dd1-4445-bd69-4c19f184559d-kube-api-access-8xlp4\") pod \"payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl\" (UID: \"5ae067e1-9dd1-4445-bd69-4c19f184559d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"
Nov 02 09:53:57.799268 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.798235    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e13bb8-dd69-470b-b395-769e72deb289-plugins-config-volume\") pod \"payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2\" (UID: \"b4e13bb8-dd69-470b-b395-769e72deb289\") " pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"
Nov 02 09:53:57.799268 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.798244    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae067e1-9dd1-4445-bd69-4c19f184559d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl\" (UID: \"5ae067e1-9dd1-4445-bd69-4c19f184559d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"
Nov 02 09:53:57.810931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.810900    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xlp4\" (UniqueName: \"kubernetes.io/projected/5ae067e1-9dd1-4445-bd69-4c19f184559d-kube-api-access-8xlp4\") pod \"payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl\" (UID: \"5ae067e1-9dd1-4445-bd69-4c19f184559d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"
Nov 02 09:53:57.811594 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.811574    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5s2v\" (UniqueName: \"kubernetes.io/projected/b4e13bb8-dd69-470b-b395-769e72deb289-kube-api-access-b5s2v\") pod \"payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2\" (UID: \"b4e13bb8-dd69-470b-b395-769e72deb289\") " pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"
Nov 02 09:53:57.881818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:57.881732    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"
Nov 02 09:53:58.060332 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:58.060296    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"
Nov 02 09:53:58.171802 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:53:58.171748    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae067e1_9dd1_4445_bd69_4c19f184559d.slice/crio-0a3840180d32d5779b73676cd05571b48f269bcf11e709146824538aff91878b WatchSource:0}: Error finding container 0a3840180d32d5779b73676cd05571b48f269bcf11e709146824538aff91878b: Status 404 returned error can't find the container with id 0a3840180d32d5779b73676cd05571b48f269bcf11e709146824538aff91878b
Nov 02 09:53:58.174194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:58.174168    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"]
Nov 02 09:53:58.347758 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:53:58.347723    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e13bb8_dd69_470b_b395_769e72deb289.slice/crio-3a265c0fe0d0a4dad55b1e9de72c037172c74300278bbe2976decc7249497cd4 WatchSource:0}: Error finding container 3a265c0fe0d0a4dad55b1e9de72c037172c74300278bbe2976decc7249497cd4: Status 404 returned error can't find the container with id 3a265c0fe0d0a4dad55b1e9de72c037172c74300278bbe2976decc7249497cd4
Nov 02 09:53:58.349089 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:58.349065    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"]
Nov 02 09:53:58.612265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:58.612210    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2" event={"ID":"b4e13bb8-dd69-470b-b395-769e72deb289","Type":"ContainerStarted","Data":"17285b3dfa55b15d3fc46261a69588bbd2e9e038423b50e6a1f77afd1b99ef5e"}
Nov 02 09:53:58.612769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:58.612292    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2" event={"ID":"b4e13bb8-dd69-470b-b395-769e72deb289","Type":"ContainerStarted","Data":"3a265c0fe0d0a4dad55b1e9de72c037172c74300278bbe2976decc7249497cd4"}
Nov 02 09:53:58.612769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:58.612451    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"
Nov 02 09:53:58.614129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:58.614105    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl" event={"ID":"5ae067e1-9dd1-4445-bd69-4c19f184559d","Type":"ContainerStarted","Data":"11a611ab5ff08d5b2acc171a24bfa1e1b07ce35c485f8473ba0687f9deae0bc4"}
Nov 02 09:53:58.614250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:58.614145    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl" event={"ID":"5ae067e1-9dd1-4445-bd69-4c19f184559d","Type":"ContainerStarted","Data":"0a3840180d32d5779b73676cd05571b48f269bcf11e709146824538aff91878b"}
Nov 02 09:53:58.614250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:58.614180    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"
Nov 02 09:53:58.634194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:58.634147    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2" podStartSLOduration=1.6341348980000001 podStartE2EDuration="1.634134898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:58.63196041 +0000 UTC m=+1115.570247186" watchObservedRunningTime="2026-11-02 09:53:58.634134898 +0000 UTC m=+1115.572421706"
Nov 02 09:53:58.663310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:58.663251    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl" podStartSLOduration=1.6632378 podStartE2EDuration="1.6632378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:58.657540596 +0000 UTC m=+1115.595827377" watchObservedRunningTime="2026-11-02 09:53:58.6632378 +0000 UTC m=+1115.601524575"
Nov 02 09:53:59.211702 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:53:59.211665    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af7de5d-8b4e-43b3-ab61-82bdd06b62bf" path="/var/lib/kubelet/pods/6af7de5d-8b4e-43b3-ab61-82bdd06b62bf/volumes"
Nov 02 09:54:01.081974 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.081710    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"]
Nov 02 09:54:01.137241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.133352    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"
Nov 02 09:54:01.156244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.153089    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:54:01.160237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.159350    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:54:01.160237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.159993    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:54:01.185241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.179073    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:54:01.272111 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.270985    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"]
Nov 02 09:54:01.370298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.369759    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d1fa4ab-dl5mf\" (UID: \"44bc7caa-a813-480b-adb7-e7a41fb91d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"
Nov 02 09:54:01.370298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.370107    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmjzc\" (UniqueName: \"kubernetes.io/projected/44bc7caa-a813-480b-adb7-e7a41fb91d99-kube-api-access-tmjzc\") pod \"maas-api-revoke-sub-3d1fa4ab-dl5mf\" (UID: \"44bc7caa-a813-480b-adb7-e7a41fb91d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"
Nov 02 09:54:01.374239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.370745    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d1fa4ab-dl5mf\" (UID: \"44bc7caa-a813-480b-adb7-e7a41fb91d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"
Nov 02 09:54:01.478854 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.473043    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjzc\" (UniqueName: \"kubernetes.io/projected/44bc7caa-a813-480b-adb7-e7a41fb91d99-kube-api-access-tmjzc\") pod \"maas-api-revoke-sub-3d1fa4ab-dl5mf\" (UID: \"44bc7caa-a813-480b-adb7-e7a41fb91d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"
Nov 02 09:54:01.478854 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.473650    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d1fa4ab-dl5mf\" (UID: \"44bc7caa-a813-480b-adb7-e7a41fb91d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"
Nov 02 09:54:01.478854 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.473802    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d1fa4ab-dl5mf\" (UID: \"44bc7caa-a813-480b-adb7-e7a41fb91d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"
Nov 02 09:54:01.497240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.483946    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d1fa4ab-dl5mf\" (UID: \"44bc7caa-a813-480b-adb7-e7a41fb91d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"
Nov 02 09:54:01.537244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.521789    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d1fa4ab-dl5mf\" (UID: \"44bc7caa-a813-480b-adb7-e7a41fb91d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"
Nov 02 09:54:01.537913 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.537881    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjzc\" (UniqueName: \"kubernetes.io/projected/44bc7caa-a813-480b-adb7-e7a41fb91d99-kube-api-access-tmjzc\") pod \"maas-api-revoke-sub-3d1fa4ab-dl5mf\" (UID: \"44bc7caa-a813-480b-adb7-e7a41fb91d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"
Nov 02 09:54:01.565241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:01.564482    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"
Nov 02 09:54:02.437490 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:02.437386    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"]
Nov 02 09:54:02.740774 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:02.740686    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf" event={"ID":"44bc7caa-a813-480b-adb7-e7a41fb91d99","Type":"ContainerStarted","Data":"fac6beee162daf54188ef05b34411cc0c6c685bd74615162c3454f2c5d0afd0d"}
Nov 02 09:54:03.183406 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.182562    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"]
Nov 02 09:54:03.189598 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.188712    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"
Nov 02 09:54:03.198135 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.198084    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed5058e-bb88-45a0-a231-198f4214bf7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7\" (UID: \"5ed5058e-bb88-45a0-a231-198f4214bf7c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"
Nov 02 09:54:03.198476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.198450    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9cbd331c\""
Nov 02 09:54:03.199327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.199306    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9cbd331c-dockercfg-jt4t2\""
Nov 02 09:54:03.200861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.198639    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnp8\" (UniqueName: \"kubernetes.io/projected/5ed5058e-bb88-45a0-a231-198f4214bf7c-kube-api-access-bvnp8\") pod \"payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7\" (UID: \"5ed5058e-bb88-45a0-a231-198f4214bf7c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"
Nov 02 09:54:03.203590 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.198725    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"]
Nov 02 09:54:03.281250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.264356    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"]
Nov 02 09:54:03.400867 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.400749    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed5058e-bb88-45a0-a231-198f4214bf7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7\" (UID: \"5ed5058e-bb88-45a0-a231-198f4214bf7c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"
Nov 02 09:54:03.401412 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.401346    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnp8\" (UniqueName: \"kubernetes.io/projected/5ed5058e-bb88-45a0-a231-198f4214bf7c-kube-api-access-bvnp8\") pod \"payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7\" (UID: \"5ed5058e-bb88-45a0-a231-198f4214bf7c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"
Nov 02 09:54:03.402210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.402181    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed5058e-bb88-45a0-a231-198f4214bf7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7\" (UID: \"5ed5058e-bb88-45a0-a231-198f4214bf7c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"
Nov 02 09:54:03.407018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.406983    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"]
Nov 02 09:54:03.407157 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.407116    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"
Nov 02 09:54:03.416527 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.416504    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnp8\" (UniqueName: \"kubernetes.io/projected/5ed5058e-bb88-45a0-a231-198f4214bf7c-kube-api-access-bvnp8\") pod \"payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7\" (UID: \"5ed5058e-bb88-45a0-a231-198f4214bf7c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"
Nov 02 09:54:03.504924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.504892    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9mzs\" (UniqueName: \"kubernetes.io/projected/12cade1c-2ef3-4274-88ef-fe9686d7064d-kube-api-access-h9mzs\") pod \"payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn\" (UID: \"12cade1c-2ef3-4274-88ef-fe9686d7064d\") " pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"
Nov 02 09:54:03.505416 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.505140    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cade1c-2ef3-4274-88ef-fe9686d7064d-plugins-config-volume\") pod \"payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn\" (UID: \"12cade1c-2ef3-4274-88ef-fe9686d7064d\") " pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"
Nov 02 09:54:03.606698 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.606663    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9mzs\" (UniqueName: \"kubernetes.io/projected/12cade1c-2ef3-4274-88ef-fe9686d7064d-kube-api-access-h9mzs\") pod \"payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn\" (UID: \"12cade1c-2ef3-4274-88ef-fe9686d7064d\") " pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"
Nov 02 09:54:03.606862 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.606840    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cade1c-2ef3-4274-88ef-fe9686d7064d-plugins-config-volume\") pod \"payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn\" (UID: \"12cade1c-2ef3-4274-88ef-fe9686d7064d\") " pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"
Nov 02 09:54:03.607609 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.607579    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cade1c-2ef3-4274-88ef-fe9686d7064d-plugins-config-volume\") pod \"payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn\" (UID: \"12cade1c-2ef3-4274-88ef-fe9686d7064d\") " pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"
Nov 02 09:54:03.620194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.620168    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9mzs\" (UniqueName: \"kubernetes.io/projected/12cade1c-2ef3-4274-88ef-fe9686d7064d-kube-api-access-h9mzs\") pod \"payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn\" (UID: \"12cade1c-2ef3-4274-88ef-fe9686d7064d\") " pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"
Nov 02 09:54:03.701480 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.700929    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"
Nov 02 09:54:03.751462 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.746639    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"
Nov 02 09:54:03.907199 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.907166    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"]
Nov 02 09:54:03.919232 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.915484    4449 generic.go:356] "Generic (PLEG): container finished" podID="44bc7caa-a813-480b-adb7-e7a41fb91d99" containerID="177c7ebd69ab47d557c341f66a71146b3d8a40cc30998049800dfe51d6f64857" exitCode=0
Nov 02 09:54:03.922358 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.922329    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"]
Nov 02 09:54:03.922571 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.922549    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf" event={"ID":"44bc7caa-a813-480b-adb7-e7a41fb91d99","Type":"ContainerDied","Data":"177c7ebd69ab47d557c341f66a71146b3d8a40cc30998049800dfe51d6f64857"}
Nov 02 09:54:03.922723 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:03.922454    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.042044 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.042003    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.042360 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.042334    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48zth\" (UniqueName: \"kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-kube-api-access-48zth\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.042485 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.042429    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.042836 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.042694    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.075916 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.075879    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"]
Nov 02 09:54:04.081195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.081164    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9a8d23-tkkq9"]
Nov 02 09:54:04.143864 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.143824    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.144063 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.143929    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.144136 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.144070    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48zth\" (UniqueName: \"kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-kube-api-access-48zth\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.144190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.144161    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.144608 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.144574    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.144952 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.144929    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.147383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.147353    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.198343 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.198289    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48zth\" (UniqueName: \"kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-kube-api-access-48zth\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.226657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.226550    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"]
Nov 02 09:54:04.236933 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:54:04.236881    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cade1c_2ef3_4274_88ef_fe9686d7064d.slice/crio-7008e3e5076c7aaa210f68aa6be47f94e13fa9bad6833e7245c2b3d14f9b085f WatchSource:0}: Error finding container 7008e3e5076c7aaa210f68aa6be47f94e13fa9bad6833e7245c2b3d14f9b085f: Status 404 returned error can't find the container with id 7008e3e5076c7aaa210f68aa6be47f94e13fa9bad6833e7245c2b3d14f9b085f
Nov 02 09:54:04.250494 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.250463    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"]
Nov 02 09:54:04.293688 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.293608    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.297540 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:54:04.297511    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed5058e_bb88_45a0_a231_198f4214bf7c.slice/crio-adbca93c3c9b9bf390d6862955d8a0c677785a5a743d5997dfd4a575bffc7903 WatchSource:0}: Error finding container adbca93c3c9b9bf390d6862955d8a0c677785a5a743d5997dfd4a575bffc7903: Status 404 returned error can't find the container with id adbca93c3c9b9bf390d6862955d8a0c677785a5a743d5997dfd4a575bffc7903
Nov 02 09:54:04.628681 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:54:04.628646    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1af3d18_a356_48bd_9b1d_ccd78a76e4e0.slice/crio-570465c8eaba0ff7ac1cb916103ce98a5d102981f52f2c97b0ff40643bec6401 WatchSource:0}: Error finding container 570465c8eaba0ff7ac1cb916103ce98a5d102981f52f2c97b0ff40643bec6401: Status 404 returned error can't find the container with id 570465c8eaba0ff7ac1cb916103ce98a5d102981f52f2c97b0ff40643bec6401
Nov 02 09:54:04.629495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.629367    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"]
Nov 02 09:54:04.929070 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.928974    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7" event={"ID":"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0","Type":"ContainerStarted","Data":"d24e78933907ddae3fdb3aff67fb8b3c2b7adfa1964826ca6ae7ae289e9b1326"}
Nov 02 09:54:04.929070 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.929033    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7" event={"ID":"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0","Type":"ContainerStarted","Data":"570465c8eaba0ff7ac1cb916103ce98a5d102981f52f2c97b0ff40643bec6401"}
Nov 02 09:54:04.929338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.929073    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:04.931009 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.930979    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn" event={"ID":"12cade1c-2ef3-4274-88ef-fe9686d7064d","Type":"ContainerStarted","Data":"53dd3108f1efb67ec157fbe2c9a90c8654d337541006f4c4f90912d8dd983a76"}
Nov 02 09:54:04.931163 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.931028    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"
Nov 02 09:54:04.931163 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.931048    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn" event={"ID":"12cade1c-2ef3-4274-88ef-fe9686d7064d","Type":"ContainerStarted","Data":"7008e3e5076c7aaa210f68aa6be47f94e13fa9bad6833e7245c2b3d14f9b085f"}
Nov 02 09:54:04.932813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.932791    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7" event={"ID":"5ed5058e-bb88-45a0-a231-198f4214bf7c","Type":"ContainerStarted","Data":"e9424cab4f6d2f3bdf195cd5b9e3275e7913bf2144469d5fad9c2e9efef745ea"}
Nov 02 09:54:04.932927 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.932826    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7" event={"ID":"5ed5058e-bb88-45a0-a231-198f4214bf7c","Type":"ContainerStarted","Data":"adbca93c3c9b9bf390d6862955d8a0c677785a5a743d5997dfd4a575bffc7903"}
Nov 02 09:54:04.932927 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.932877    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"
Nov 02 09:54:04.951288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.951205    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7" podStartSLOduration=1.951188786 podStartE2EDuration="1.951188786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:04.948396301 +0000 UTC m=+1121.886683095" watchObservedRunningTime="2026-11-02 09:54:04.951188786 +0000 UTC m=+1121.889475562"
Nov 02 09:54:04.973929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.973878    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7" podStartSLOduration=1.973864352 podStartE2EDuration="1.973864352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:04.967791857 +0000 UTC m=+1121.906078662" watchObservedRunningTime="2026-11-02 09:54:04.973864352 +0000 UTC m=+1121.912151128"
Nov 02 09:54:04.988846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:04.988787    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn" podStartSLOduration=1.9887645109999998 podStartE2EDuration="1.988764511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:04.988311962 +0000 UTC m=+1121.926598768" watchObservedRunningTime="2026-11-02 09:54:04.988764511 +0000 UTC m=+1121.927051289"
Nov 02 09:54:05.101930 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.101878    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"
Nov 02 09:54:05.160767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.160732    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-xks-ca\") pod \"44bc7caa-a813-480b-adb7-e7a41fb91d99\" (UID: \"44bc7caa-a813-480b-adb7-e7a41fb91d99\") "
Nov 02 09:54:05.160975 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.160814    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-openshift-ca\") pod \"44bc7caa-a813-480b-adb7-e7a41fb91d99\" (UID: \"44bc7caa-a813-480b-adb7-e7a41fb91d99\") "
Nov 02 09:54:05.160975 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.160904    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44bc7caa-a813-480b-adb7-e7a41fb91d99-kube-api-access-tmjzc\" (UniqueName: \"kubernetes.io/projected/44bc7caa-a813-480b-adb7-e7a41fb91d99-kube-api-access-tmjzc\") pod \"44bc7caa-a813-480b-adb7-e7a41fb91d99\" (UID: \"44bc7caa-a813-480b-adb7-e7a41fb91d99\") "
Nov 02 09:54:05.161318 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.161287    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-openshift-ca" pod "44bc7caa-a813-480b-adb7-e7a41fb91d99" (UID: "44bc7caa-a813-480b-adb7-e7a41fb91d99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:05.161436 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.161414    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:05.164002 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.163966    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-xks-ca" pod "44bc7caa-a813-480b-adb7-e7a41fb91d99" (UID: "44bc7caa-a813-480b-adb7-e7a41fb91d99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:54:05.165516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.165487    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44bc7caa-a813-480b-adb7-e7a41fb91d99-kube-api-access-tmjzc" pod "44bc7caa-a813-480b-adb7-e7a41fb91d99" (UID: "44bc7caa-a813-480b-adb7-e7a41fb91d99"). InnerVolumeSpecName "kube-api-access-tmjzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:05.209408 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.209326    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695c1281-51d3-45c8-85f2-04838c8bce77" path="/var/lib/kubelet/pods/695c1281-51d3-45c8-85f2-04838c8bce77/volumes"
Nov 02 09:54:05.262454 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.262424    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmjzc\" (UniqueName: \"kubernetes.io/projected/44bc7caa-a813-480b-adb7-e7a41fb91d99-kube-api-access-tmjzc\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:05.262454 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.262454    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bc7caa-a813-480b-adb7-e7a41fb91d99-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:05.944547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.944470    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"
Nov 02 09:54:05.944547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.944485    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf" event={"ID":"44bc7caa-a813-480b-adb7-e7a41fb91d99","Type":"ContainerDied","Data":"fac6beee162daf54188ef05b34411cc0c6c685bd74615162c3454f2c5d0afd0d"}
Nov 02 09:54:05.944547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:05.944544    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fac6beee162daf54188ef05b34411cc0c6c685bd74615162c3454f2c5d0afd0d"
Nov 02 09:54:06.548210 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:06.548171    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:54:06.568563 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:06.568541    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"
Nov 02 09:54:08.032107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.032062    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"]
Nov 02 09:54:08.067466 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.067417    4449 cpu_manager.go:414] "containerMap: removing container" podUID="44bc7caa-a813-480b-adb7-e7a41fb91d99" containerName="revoke-keys"
Nov 02 09:54:08.067466 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.067453    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44bc7caa-a813-480b-adb7-e7a41fb91d99" containerName="revoke-keys"
Nov 02 09:54:08.067870 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.067845    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="44bc7caa-a813-480b-adb7-e7a41fb91d99" containerName="revoke-keys"
Nov 02 09:54:08.074455 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.074428    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"]
Nov 02 09:54:08.074609 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.074543    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.120239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.120183    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.120434 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.120364    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.120434 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.120399    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.120575 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.120503    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.120575 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.120546    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.120682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.120638    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctvg8\" (UniqueName: \"kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-kube-api-access-ctvg8\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.120741 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.120710    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.120803 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.120780    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.120919 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.120904    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.222875 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.222833    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.223080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.222883    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.223080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.222933    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.223080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.222964    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.223080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.223018    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctvg8\" (UniqueName: \"kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-kube-api-access-ctvg8\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.223080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.223066    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.223397 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.223103    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.223397 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.223256    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.223502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.223409    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.223502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.223471    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.223599 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.223529    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.223764 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.223740    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.223869 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.223839    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.223941 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.223918    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.225541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.225518    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.225858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.225838    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.235666 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.235644    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.235844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.235825    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctvg8\" (UniqueName: \"kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-kube-api-access-ctvg8\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.394899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.394859    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:08.692558 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:54:08.692525    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bccd9c_ca7e_4f26_81da_3d5d77be08df.slice/crio-3c3a524ccb280649dfdd1d3ed951ca08a5b32c70350002391bde8a06dfbe5329 WatchSource:0}: Error finding container 3c3a524ccb280649dfdd1d3ed951ca08a5b32c70350002391bde8a06dfbe5329: Status 404 returned error can't find the container with id 3c3a524ccb280649dfdd1d3ed951ca08a5b32c70350002391bde8a06dfbe5329
Nov 02 09:54:08.695341 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.695301    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:08.695475 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.695387    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:08.695475 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.695434    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:08.695623 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.695597    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"]
Nov 02 09:54:08.987088 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.987003    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" event={"ID":"a1bccd9c-ca7e-4f26-81da-3d5d77be08df","Type":"ContainerStarted","Data":"ce4537aa0227c6323fb9eb3be763dfcca4cd680f5785062d98390f41ccfaf626"}
Nov 02 09:54:08.987088 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:08.987057    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" event={"ID":"a1bccd9c-ca7e-4f26-81da-3d5d77be08df","Type":"ContainerStarted","Data":"3c3a524ccb280649dfdd1d3ed951ca08a5b32c70350002391bde8a06dfbe5329"}
Nov 02 09:54:09.019629 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:09.019581    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" podStartSLOduration=2.019566411 podStartE2EDuration="2.019566411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:09.01533062 +0000 UTC m=+1125.953617397" watchObservedRunningTime="2026-11-02 09:54:09.019566411 +0000 UTC m=+1125.957853186"
Nov 02 09:54:09.395508 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:09.395285    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:09.396870 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:09.396835    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 02 09:54:09.396992 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:09.396896    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 02 09:54:09.628242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:09.628185    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"
Nov 02 09:54:09.628427 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:09.628355    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"
Nov 02 09:54:10.397241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:10.396427    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 02 09:54:10.397241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:10.396502    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 02 09:54:11.395833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:11.395797    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 02 09:54:11.396022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:11.395870    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 02 09:54:11.790672 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:11.790622    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"]
Nov 02 09:54:11.795726 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:11.795700    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"
Nov 02 09:54:11.800687 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:11.800644    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:54:11.801767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:11.801736    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:54:11.801767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:11.801762    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:54:11.802376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:11.802354    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:54:11.806100 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:11.806064    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"]
Nov 02 09:54:11.922945 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:11.922896    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51d514f9-jmlmm\" (UID: \"b72589ec-4fb0-4348-a63f-57178b3623e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"
Nov 02 09:54:11.923117 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:11.922986    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wmc9\" (UniqueName: \"kubernetes.io/projected/b72589ec-4fb0-4348-a63f-57178b3623e3-kube-api-access-4wmc9\") pod \"maas-api-revoke-sub-51d514f9-jmlmm\" (UID: \"b72589ec-4fb0-4348-a63f-57178b3623e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"
Nov 02 09:54:11.923117 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:11.923045    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51d514f9-jmlmm\" (UID: \"b72589ec-4fb0-4348-a63f-57178b3623e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"
Nov 02 09:54:12.024281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.024201    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51d514f9-jmlmm\" (UID: \"b72589ec-4fb0-4348-a63f-57178b3623e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"
Nov 02 09:54:12.024479 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.024301    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wmc9\" (UniqueName: \"kubernetes.io/projected/b72589ec-4fb0-4348-a63f-57178b3623e3-kube-api-access-4wmc9\") pod \"maas-api-revoke-sub-51d514f9-jmlmm\" (UID: \"b72589ec-4fb0-4348-a63f-57178b3623e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"
Nov 02 09:54:12.024479 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.024340    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51d514f9-jmlmm\" (UID: \"b72589ec-4fb0-4348-a63f-57178b3623e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"
Nov 02 09:54:12.025095 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.025071    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51d514f9-jmlmm\" (UID: \"b72589ec-4fb0-4348-a63f-57178b3623e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"
Nov 02 09:54:12.025529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.025504    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"]
Nov 02 09:54:12.027241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.027199    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51d514f9-jmlmm\" (UID: \"b72589ec-4fb0-4348-a63f-57178b3623e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"
Nov 02 09:54:12.030767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.030742    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.040915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.040836    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wmc9\" (UniqueName: \"kubernetes.io/projected/b72589ec-4fb0-4348-a63f-57178b3623e3-kube-api-access-4wmc9\") pod \"maas-api-revoke-sub-51d514f9-jmlmm\" (UID: \"b72589ec-4fb0-4348-a63f-57178b3623e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"
Nov 02 09:54:12.046438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.046403    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"]
Nov 02 09:54:12.149921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.133708    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"
Nov 02 09:54:12.149921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.135343    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h977d\" (UniqueName: \"kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-kube-api-access-h977d\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.149921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.135404    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.149921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.135551    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.149921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.135604    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.149921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.135649    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.149921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.135677    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c2db3f-6464-47e8-af64-2c432474e832-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.149921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.135793    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c2db3f-6464-47e8-af64-2c432474e832-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.149921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.135832    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.149921 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.135865    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.239169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.237512    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h977d\" (UniqueName: \"kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-kube-api-access-h977d\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.239169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.237569    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.239169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.237672    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.239169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.237706    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.239169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.237740    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.239169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.237759    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c2db3f-6464-47e8-af64-2c432474e832-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.239169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.237838    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c2db3f-6464-47e8-af64-2c432474e832-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.239169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.237866    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.239169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.237893    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.239169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.238405    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.239169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.238893    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.244277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.241401    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.244277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.242123    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c2db3f-6464-47e8-af64-2c432474e832-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.244277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.244228    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.246011 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.245957    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.248027 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.247988    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c2db3f-6464-47e8-af64-2c432474e832-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.291476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.291377    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.291790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.291762    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h977d\" (UniqueName: \"kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-kube-api-access-h977d\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.291899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.291879    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:12.397452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.396602    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 02 09:54:12.397452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.396661    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 02 09:54:12.525471 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:54:12.525443    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72589ec_4fb0_4348_a63f_57178b3623e3.slice/crio-1136a63add057be15601d4fc4384cb1ab02cd1378f0aaf3ba7259f3435bbcc95 WatchSource:0}: Error finding container 1136a63add057be15601d4fc4384cb1ab02cd1378f0aaf3ba7259f3435bbcc95: Status 404 returned error can't find the container with id 1136a63add057be15601d4fc4384cb1ab02cd1378f0aaf3ba7259f3435bbcc95
Nov 02 09:54:12.527140 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.527111    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"]
Nov 02 09:54:12.645980 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.645942    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:12.646075 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.646019    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:12.646075 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.646050    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:12.646303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:12.646283    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"]
Nov 02 09:54:13.027461 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:13.027414    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" event={"ID":"11c2db3f-6464-47e8-af64-2c432474e832","Type":"ContainerStarted","Data":"20bc5977a89804e49285c5d35b26f3dd0ff7a62db9534c2336e1c4914add7854"}
Nov 02 09:54:13.027890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:13.027493    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" event={"ID":"11c2db3f-6464-47e8-af64-2c432474e832","Type":"ContainerStarted","Data":"7df62c34e349721a5314dabfbc0c20b54d24d1a6d2f6e289c8f8488ebd9eb09b"}
Nov 02 09:54:13.029524 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:13.029490    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm" event={"ID":"b72589ec-4fb0-4348-a63f-57178b3623e3","Type":"ContainerStarted","Data":"1136a63add057be15601d4fc4384cb1ab02cd1378f0aaf3ba7259f3435bbcc95"}
Nov 02 09:54:13.083040 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:13.082980    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" podStartSLOduration=1.082963191 podStartE2EDuration="1.082963191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:13.07750118 +0000 UTC m=+1130.015787982" watchObservedRunningTime="2026-11-02 09:54:13.082963191 +0000 UTC m=+1130.021249966"
Nov 02 09:54:13.293149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:13.293118    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:13.294401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:13.294376    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 09:54:13.294523 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:13.294431    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" podUID="11c2db3f-6464-47e8-af64-2c432474e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 09:54:13.395425 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:13.395389    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 02 09:54:13.395601 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:13.395446    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 02 09:54:14.043571 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:14.043541    4449 generic.go:356] "Generic (PLEG): container finished" podID="b72589ec-4fb0-4348-a63f-57178b3623e3" containerID="66e93384ea42b4c71d908e7983d8ab4fb5bac2ba93052bd115d57b755310a453" exitCode=0
Nov 02 09:54:14.044183 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:14.043722    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm" event={"ID":"b72589ec-4fb0-4348-a63f-57178b3623e3","Type":"ContainerDied","Data":"66e93384ea42b4c71d908e7983d8ab4fb5bac2ba93052bd115d57b755310a453"}
Nov 02 09:54:14.299301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:14.298375    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 09:54:14.299301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:14.298441    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" podUID="11c2db3f-6464-47e8-af64-2c432474e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 09:54:14.402248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:14.401592    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 02 09:54:14.402248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:14.401657    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 02 09:54:15.292782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.292753    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 09:54:15.293248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.292803    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" podUID="11c2db3f-6464-47e8-af64-2c432474e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 09:54:15.317571 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.317544    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"
Nov 02 09:54:15.395702 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.395608    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 02 09:54:15.395977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.395941    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 02 09:54:15.438779 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.438740    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72589ec-4fb0-4348-a63f-57178b3623e3-kube-api-access-4wmc9\" (UniqueName: \"kubernetes.io/projected/b72589ec-4fb0-4348-a63f-57178b3623e3-kube-api-access-4wmc9\") pod \"b72589ec-4fb0-4348-a63f-57178b3623e3\" (UID: \"b72589ec-4fb0-4348-a63f-57178b3623e3\") "
Nov 02 09:54:15.438954 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.438832    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-xks-ca\") pod \"b72589ec-4fb0-4348-a63f-57178b3623e3\" (UID: \"b72589ec-4fb0-4348-a63f-57178b3623e3\") "
Nov 02 09:54:15.439115 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.439097    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-openshift-ca\") pod \"b72589ec-4fb0-4348-a63f-57178b3623e3\" (UID: \"b72589ec-4fb0-4348-a63f-57178b3623e3\") "
Nov 02 09:54:15.439594 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.439564    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-openshift-ca" pod "b72589ec-4fb0-4348-a63f-57178b3623e3" (UID: "b72589ec-4fb0-4348-a63f-57178b3623e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:15.440119 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.440082    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:15.487243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.478596    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72589ec-4fb0-4348-a63f-57178b3623e3-kube-api-access-4wmc9" pod "b72589ec-4fb0-4348-a63f-57178b3623e3" (UID: "b72589ec-4fb0-4348-a63f-57178b3623e3"). InnerVolumeSpecName "kube-api-access-4wmc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:15.487461 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.486431    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-xks-ca" pod "b72589ec-4fb0-4348-a63f-57178b3623e3" (UID: "b72589ec-4fb0-4348-a63f-57178b3623e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:54:15.541871 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.541836    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wmc9\" (UniqueName: \"kubernetes.io/projected/b72589ec-4fb0-4348-a63f-57178b3623e3-kube-api-access-4wmc9\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:15.541871 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.541867    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72589ec-4fb0-4348-a63f-57178b3623e3-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:15.987252 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.986090    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"
Nov 02 09:54:15.992241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:15.988738    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"
Nov 02 09:54:16.010243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.004449    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:16.130050 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:16.130004    4449 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-4a99520c97db: secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:16.130050 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:16.130046    4449 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7: secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:16.135931 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:16.135895    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials podName:f1af3d18-a356-48bd-9b1d-ccd78a76e4e0 nodeName:}" failed. No retries permitted until 2026-11-02 09:54:16.635870131 +0000 UTC m=+1133.574156898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7" (UID: "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0") : secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:16.240299 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.240187    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm" event={"ID":"b72589ec-4fb0-4348-a63f-57178b3623e3","Type":"ContainerDied","Data":"1136a63add057be15601d4fc4384cb1ab02cd1378f0aaf3ba7259f3435bbcc95"}
Nov 02 09:54:16.240548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.240532    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1136a63add057be15601d4fc4384cb1ab02cd1378f0aaf3ba7259f3435bbcc95"
Nov 02 09:54:16.243999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.243978    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"
Nov 02 09:54:16.293050 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.292982    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 09:54:16.293674 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.293642    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" podUID="11c2db3f-6464-47e8-af64-2c432474e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 09:54:16.651392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.651354    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"]
Nov 02 09:54:16.680572 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:16.680535    4449 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-4a99520c97db: secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:16.680784 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:16.680767    4449 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7: secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:16.680957 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:16.680944    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials podName:f1af3d18-a356-48bd-9b1d-ccd78a76e4e0 nodeName:}" failed. No retries permitted until 2026-11-02 09:54:17.680920788 +0000 UTC m=+1134.619207556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7" (UID: "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0") : secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:16.729431 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.728446    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"]
Nov 02 09:54:16.730411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.730130    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b72589ec-4fb0-4348-a63f-57178b3623e3" containerName="revoke-keys"
Nov 02 09:54:16.730411 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.730159    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72589ec-4fb0-4348-a63f-57178b3623e3" containerName="revoke-keys"
Nov 02 09:54:16.731109 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.731003    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72589ec-4fb0-4348-a63f-57178b3623e3" containerName="revoke-keys"
Nov 02 09:54:16.735972 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.735952    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"]
Nov 02 09:54:16.736190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.736174    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:16.744658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.744635    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"]
Nov 02 09:54:16.786900 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.755452    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"
Nov 02 09:54:16.787255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.786345    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"]
Nov 02 09:54:16.797293 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.797175    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:54:16.797506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.797477    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:54:16.797623 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.797573    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:54:16.797623 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.797373    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:54:16.798600 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.798565    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw\" (UID: \"3f619613-433a-4f61-827b-9fdc47641b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"
Nov 02 09:54:16.798703 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.798638    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw\" (UID: \"3f619613-433a-4f61-827b-9fdc47641b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"
Nov 02 09:54:16.798769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.798749    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:16.798818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.798787    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn5dp\" (UniqueName: \"kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-kube-api-access-jn5dp\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:16.798882 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.798823    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:16.798882 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.798853    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmtj2\" (UniqueName: \"kubernetes.io/projected/3f619613-433a-4f61-827b-9fdc47641b23-kube-api-access-vmtj2\") pod \"maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw\" (UID: \"3f619613-433a-4f61-827b-9fdc47641b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"
Nov 02 09:54:16.799474 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.799446    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:16.810537 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.808905    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"]
Nov 02 09:54:16.812388 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.810686    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"
Nov 02 09:54:16.885383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.885355    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"]
Nov 02 09:54:16.915292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.909242    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:16.915292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.909300    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn5dp\" (UniqueName: \"kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-kube-api-access-jn5dp\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:16.915292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.909341    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:16.915292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.909374    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmtj2\" (UniqueName: \"kubernetes.io/projected/3f619613-433a-4f61-827b-9fdc47641b23-kube-api-access-vmtj2\") pod \"maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw\" (UID: \"3f619613-433a-4f61-827b-9fdc47641b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"
Nov 02 09:54:16.915292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.909556    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:16.915292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.909658    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw\" (UID: \"3f619613-433a-4f61-827b-9fdc47641b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"
Nov 02 09:54:16.915292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.909717    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw\" (UID: \"3f619613-433a-4f61-827b-9fdc47641b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"
Nov 02 09:54:16.919240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.918626    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:16.930644 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.930591    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:16.931070 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.931032    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw\" (UID: \"3f619613-433a-4f61-827b-9fdc47641b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"
Nov 02 09:54:16.979257 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.977121    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw\" (UID: \"3f619613-433a-4f61-827b-9fdc47641b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"
Nov 02 09:54:17.010263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:16.994847    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:17.053244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.041779    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2453a69-qqxdl\" (UID: \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"
Nov 02 09:54:17.053244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.044690    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmtj2\" (UniqueName: \"kubernetes.io/projected/3f619613-433a-4f61-827b-9fdc47641b23-kube-api-access-vmtj2\") pod \"maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw\" (UID: \"3f619613-433a-4f61-827b-9fdc47641b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"
Nov 02 09:54:17.077317 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.066824    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2453a69-qqxdl\" (UID: \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"
Nov 02 09:54:17.077317 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.066977    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kklmb\" (UniqueName: \"kubernetes.io/projected/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-kube-api-access-kklmb\") pod \"maas-api-revoke-sub-b2453a69-qqxdl\" (UID: \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"
Nov 02 09:54:17.106245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.104876    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn5dp\" (UniqueName: \"kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-kube-api-access-jn5dp\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:17.106245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.105866    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:17.244291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.242390    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2453a69-qqxdl\" (UID: \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"
Nov 02 09:54:17.244291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.242545    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2453a69-qqxdl\" (UID: \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"
Nov 02 09:54:17.244291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.242578    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kklmb\" (UniqueName: \"kubernetes.io/projected/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-kube-api-access-kklmb\") pod \"maas-api-revoke-sub-b2453a69-qqxdl\" (UID: \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"
Nov 02 09:54:17.247103 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.247052    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2453a69-qqxdl\" (UID: \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"
Nov 02 09:54:17.273323 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.250734    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"
Nov 02 09:54:17.309244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.304816    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 09:54:17.309244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.304883    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" podUID="11c2db3f-6464-47e8-af64-2c432474e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 09:54:17.359241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.346262    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kklmb\" (UniqueName: \"kubernetes.io/projected/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-kube-api-access-kklmb\") pod \"maas-api-revoke-sub-b2453a69-qqxdl\" (UID: \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"
Nov 02 09:54:17.378389 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.370895    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"]
Nov 02 09:54:17.404250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.395668    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"]
Nov 02 09:54:17.404250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.395722    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"]
Nov 02 09:54:17.404250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.395823    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.404250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.403530    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 09:54:17.404250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.403654    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2453a69-qqxdl\" (UID: \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"
Nov 02 09:54:17.404250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.403770    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 09:54:17.404250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.403931    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8zrbm\""
Nov 02 09:54:17.404250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.404002    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 09:54:17.409284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.408645    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:54:17.409284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.408692    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:17.480301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.480269    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1726b246-qwpvj"]
Nov 02 09:54:17.599433 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.595406    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"
Nov 02 09:54:17.727245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.708299    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mtzh\" (UniqueName: \"kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-kube-api-access-8mtzh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.727584 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.727549    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.727758 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.727741    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af78de-9573-422d-8563-4a32acc6e833-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.727991 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.727963    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.728865 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:17.728842    4449 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-4a99520c97db: secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:17.729009 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:17.728995    4449 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7: secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:17.729152 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:17.729141    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials podName:f1af3d18-a356-48bd-9b1d-ccd78a76e4e0 nodeName:}" failed. No retries permitted until 2026-11-02 09:54:19.729117355 +0000 UTC m=+1136.667404125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7" (UID: "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0") : secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:17.730162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.728208    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.737957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.737921    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.750255 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.738687    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.899402 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.892332    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.899402 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.892489    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.899402 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.892549    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.899402 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.892667    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtzh\" (UniqueName: \"kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-kube-api-access-8mtzh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.899402 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.892720    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.899402 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.892780    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af78de-9573-422d-8563-4a32acc6e833-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.899402 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.892900    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.915976 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.915936    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:17.997510 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:17.997476    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:18.023657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:18.023618    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af78de-9573-422d-8563-4a32acc6e833-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:18.025517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:18.025443    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:18.041463 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:18.026457    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:18.051297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:18.044771    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtzh\" (UniqueName: \"kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-kube-api-access-8mtzh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:18.052061 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:18.052006    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-scwch\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:18.102386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:18.102342    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:18.281244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:18.265605    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:18.299246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:18.296237    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 09:54:18.299246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:18.296297    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" podUID="11c2db3f-6464-47e8-af64-2c432474e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 09:54:18.334244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:18.329691    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n" podUID="7968e57e-6e72-4ef2-af9f-82c111d870f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:18.469412 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:18.462352    4449 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:54:18.469412 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:18.462421    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:19.097243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:19.083448    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:19.262341 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:19.211167    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:19.262341 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:19.241757    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:54:19.416675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:19.412322    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"]
Nov 02 09:54:19.636277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:19.633335    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 09:54:19.636277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:19.633400    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" podUID="11c2db3f-6464-47e8-af64-2c432474e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 09:54:19.659429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:19.657663    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63601e9d-7e10-406e-b77d-c30f26e519c1" path="/var/lib/kubelet/pods/63601e9d-7e10-406e-b77d-c30f26e519c1/volumes"
Nov 02 09:54:19.659429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:19.658194    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"]
Nov 02 09:54:19.659429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:19.658431    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"]
Nov 02 09:54:19.755244 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:19.748401    4449 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-4a99520c97db: secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:19.755244 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:19.748423    4449 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7: secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:19.755244 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:19.748491    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials podName:f1af3d18-a356-48bd-9b1d-ccd78a76e4e0 nodeName:}" failed. No retries permitted until 2026-11-02 09:54:23.748472081 +0000 UTC m=+1140.686758855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7" (UID: "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0") : secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:19.829384 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:19.820130    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"]
Nov 02 09:54:19.901066 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:19.898603    4449 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 09:54:19.960312 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:19.949058    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw" event={"ID":"3f619613-433a-4f61-827b-9fdc47641b23","Type":"ContainerStarted","Data":"afffe69afbf9b51595dc75e02be12e8e031e79e79075f2380623ef65396efd56"}
Nov 02 09:54:20.004984 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:20.004945    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw" event={"ID":"07d6f736-47e3-4f7f-b866-aea262f8fc3b","Type":"ContainerStarted","Data":"d20ea2220c915388a746d68a672cb01b73b9c52d40a69b62fe38e0df00448d23"}
Nov 02 09:54:20.005264 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:20.005237    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw" event={"ID":"07d6f736-47e3-4f7f-b866-aea262f8fc3b","Type":"ContainerStarted","Data":"cd539d55869dbcc617ada06e85cd8a36d539912b88a564ccf3f29352b39c9c47"}
Nov 02 09:54:20.005467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:20.005453    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:20.008048 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:20.008016    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch" event={"ID":"d1af78de-9573-422d-8563-4a32acc6e833","Type":"ContainerStarted","Data":"23d8c56256edc2db2b0a8a33381fbf1965a9200d411802dd3e7408be695e5bf4"}
Nov 02 09:54:20.011349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:20.011317    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl" event={"ID":"7e675a17-aed8-48b0-96c9-d9ccfd5c3121","Type":"ContainerStarted","Data":"783b73939dac0a9bd4bbe666252d5e4eda1e9a403523f4688710647e3f53d777"}
Nov 02 09:54:20.026303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:20.026259    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw" podStartSLOduration=4.026241752 podStartE2EDuration="4.026241752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:20.024260445 +0000 UTC m=+1136.962547232" watchObservedRunningTime="2026-11-02 09:54:20.026241752 +0000 UTC m=+1136.964528561"
Nov 02 09:54:20.294324 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:20.294079    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 09:54:20.294324 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:20.294141    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" podUID="11c2db3f-6464-47e8-af64-2c432474e832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 09:54:21.023805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:21.023769    4449 generic.go:356] "Generic (PLEG): container finished" podID="7e675a17-aed8-48b0-96c9-d9ccfd5c3121" containerID="93f1dcf8ca6df1ce4171274c8b8244a6abce6448fa6c0cfb258341ea27ea4638" exitCode=0
Nov 02 09:54:21.024293 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:21.023859    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl" event={"ID":"7e675a17-aed8-48b0-96c9-d9ccfd5c3121","Type":"ContainerDied","Data":"93f1dcf8ca6df1ce4171274c8b8244a6abce6448fa6c0cfb258341ea27ea4638"}
Nov 02 09:54:21.026233 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:21.026189    4449 generic.go:356] "Generic (PLEG): container finished" podID="3f619613-433a-4f61-827b-9fdc47641b23" containerID="c4b7dd3d68c0552c8fdff88314ac6e5bd4b3db9a0634c721cd71d5b2aea1ffd4" exitCode=0
Nov 02 09:54:21.026382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:21.026272    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw" event={"ID":"3f619613-433a-4f61-827b-9fdc47641b23","Type":"ContainerDied","Data":"c4b7dd3d68c0552c8fdff88314ac6e5bd4b3db9a0634c721cd71d5b2aea1ffd4"}
Nov 02 09:54:21.436081 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:21.436050    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:21.436469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:21.436437    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:21.437376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:21.437356    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:54:21.679243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:21.676410    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq" podUID="6692fe0c-a649-4c5f-aeb6-bd80777cd4c4" containerName="payload-processing" containerID="cri-o://136623828a325cb459923109c3235396448a3a6bc7eecefab2422d63ec388814" gracePeriod=25
Nov 02 09:54:21.995998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:21.995971    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:54:22.030931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.030896    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-routing-overlay-volume\") pod \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") "
Nov 02 09:54:22.031437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.031010    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-kube-api-access-4vpf9\" (UniqueName: \"kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-kube-api-access-4vpf9\") pod \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") "
Nov 02 09:54:22.031437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.031117    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-plugins-config-volume\") pod \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") "
Nov 02 09:54:22.031437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.031413    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-provider-credentials\") pod \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\" (UID: \"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4\") "
Nov 02 09:54:22.031602 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.031428    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-routing-overlay-volume" pod "6692fe0c-a649-4c5f-aeb6-bd80777cd4c4" (UID: "6692fe0c-a649-4c5f-aeb6-bd80777cd4c4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:22.031602 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.031556    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-plugins-config-volume" pod "6692fe0c-a649-4c5f-aeb6-bd80777cd4c4" (UID: "6692fe0c-a649-4c5f-aeb6-bd80777cd4c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:22.032102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.032077    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:22.032265 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.032104    4449 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-routing-overlay-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:22.034446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.034415    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-kube-api-access-4vpf9" pod "6692fe0c-a649-4c5f-aeb6-bd80777cd4c4" (UID: "6692fe0c-a649-4c5f-aeb6-bd80777cd4c4"). InnerVolumeSpecName "kube-api-access-4vpf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:22.034819 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.034791    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-provider-credentials" pod "6692fe0c-a649-4c5f-aeb6-bd80777cd4c4" (UID: "6692fe0c-a649-4c5f-aeb6-bd80777cd4c4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:22.039672 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.039610    4449 generic.go:356] "Generic (PLEG): container finished" podID="6692fe0c-a649-4c5f-aeb6-bd80777cd4c4" containerID="136623828a325cb459923109c3235396448a3a6bc7eecefab2422d63ec388814" exitCode=0
Nov 02 09:54:22.040165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.039863    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"
Nov 02 09:54:22.040165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.039937    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq" event={"ID":"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4","Type":"ContainerDied","Data":"136623828a325cb459923109c3235396448a3a6bc7eecefab2422d63ec388814"}
Nov 02 09:54:22.040165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.039991    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq" event={"ID":"6692fe0c-a649-4c5f-aeb6-bd80777cd4c4","Type":"ContainerDied","Data":"2c9413fb6031dc94789005a86fb8afbc4b453be3ea5e20ba03bf91419518ea3d"}
Nov 02 09:54:22.040165 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.040020    4449 scope.go:122] "RemoveContainer" containerID="136623828a325cb459923109c3235396448a3a6bc7eecefab2422d63ec388814"
Nov 02 09:54:22.109469 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.109440    4449 scope.go:122] "RemoveContainer" containerID="136623828a325cb459923109c3235396448a3a6bc7eecefab2422d63ec388814"
Nov 02 09:54:22.110659 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:22.110592    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136623828a325cb459923109c3235396448a3a6bc7eecefab2422d63ec388814\": container with ID starting with 136623828a325cb459923109c3235396448a3a6bc7eecefab2422d63ec388814 not found: ID does not exist" containerID="136623828a325cb459923109c3235396448a3a6bc7eecefab2422d63ec388814"
Nov 02 09:54:22.110659 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.110634    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136623828a325cb459923109c3235396448a3a6bc7eecefab2422d63ec388814"} err="failed to get container status \"136623828a325cb459923109c3235396448a3a6bc7eecefab2422d63ec388814\": rpc error: code = NotFound desc = could not find container \"136623828a325cb459923109c3235396448a3a6bc7eecefab2422d63ec388814\": container with ID starting with 136623828a325cb459923109c3235396448a3a6bc7eecefab2422d63ec388814 not found: ID does not exist"
Nov 02 09:54:22.124248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.121257    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"]
Nov 02 09:54:22.138450 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.138409    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vpf9\" (UniqueName: \"kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-kube-api-access-4vpf9\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:22.138450 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.138454    4449 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4-provider-credentials\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:22.142612 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.142575    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-f899wqrmq"]
Nov 02 09:54:22.309129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.309097    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"
Nov 02 09:54:22.379490 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.379447    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"
Nov 02 09:54:22.479206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.479108    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-xks-ca\") pod \"3f619613-433a-4f61-827b-9fdc47641b23\" (UID: \"3f619613-433a-4f61-827b-9fdc47641b23\") "
Nov 02 09:54:22.479206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.479182    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-kube-api-access-kklmb\" (UniqueName: \"kubernetes.io/projected/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-kube-api-access-kklmb\") pod \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\" (UID: \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\") "
Nov 02 09:54:22.479455 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.479262    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-openshift-ca\") pod \"3f619613-433a-4f61-827b-9fdc47641b23\" (UID: \"3f619613-433a-4f61-827b-9fdc47641b23\") "
Nov 02 09:54:22.479625 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.479602    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-openshift-ca" pod "3f619613-433a-4f61-827b-9fdc47641b23" (UID: "3f619613-433a-4f61-827b-9fdc47641b23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:22.479753 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.479670    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f619613-433a-4f61-827b-9fdc47641b23-kube-api-access-vmtj2\" (UniqueName: \"kubernetes.io/projected/3f619613-433a-4f61-827b-9fdc47641b23-kube-api-access-vmtj2\") pod \"3f619613-433a-4f61-827b-9fdc47641b23\" (UID: \"3f619613-433a-4f61-827b-9fdc47641b23\") "
Nov 02 09:54:22.479813 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.479789    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-xks-ca\") pod \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\" (UID: \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\") "
Nov 02 09:54:22.479901 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.479880    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-openshift-ca\") pod \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\" (UID: \"7e675a17-aed8-48b0-96c9-d9ccfd5c3121\") "
Nov 02 09:54:22.480275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.480211    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-openshift-ca" pod "7e675a17-aed8-48b0-96c9-d9ccfd5c3121" (UID: "7e675a17-aed8-48b0-96c9-d9ccfd5c3121"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:22.480902 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.480875    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:22.481016 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.480907    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:22.481821 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.481798    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-xks-ca" pod "3f619613-433a-4f61-827b-9fdc47641b23" (UID: "3f619613-433a-4f61-827b-9fdc47641b23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:54:22.481995 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.481973    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-kube-api-access-kklmb" pod "7e675a17-aed8-48b0-96c9-d9ccfd5c3121" (UID: "7e675a17-aed8-48b0-96c9-d9ccfd5c3121"). InnerVolumeSpecName "kube-api-access-kklmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:22.482259 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.482203    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-xks-ca" pod "7e675a17-aed8-48b0-96c9-d9ccfd5c3121" (UID: "7e675a17-aed8-48b0-96c9-d9ccfd5c3121"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:54:22.482762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.482743    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f619613-433a-4f61-827b-9fdc47641b23-kube-api-access-vmtj2" pod "3f619613-433a-4f61-827b-9fdc47641b23" (UID: "3f619613-433a-4f61-827b-9fdc47641b23"). InnerVolumeSpecName "kube-api-access-vmtj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:22.582261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.582191    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:22.582261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.582260    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f619613-433a-4f61-827b-9fdc47641b23-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:22.582436 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.582273    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kklmb\" (UniqueName: \"kubernetes.io/projected/7e675a17-aed8-48b0-96c9-d9ccfd5c3121-kube-api-access-kklmb\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:22.582436 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:22.582283    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmtj2\" (UniqueName: \"kubernetes.io/projected/3f619613-433a-4f61-827b-9fdc47641b23-kube-api-access-vmtj2\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:23.081098 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:23.081070    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"
Nov 02 09:54:23.081543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:23.081103    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw" event={"ID":"3f619613-433a-4f61-827b-9fdc47641b23","Type":"ContainerDied","Data":"afffe69afbf9b51595dc75e02be12e8e031e79e79075f2380623ef65396efd56"}
Nov 02 09:54:23.081543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:23.081157    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afffe69afbf9b51595dc75e02be12e8e031e79e79075f2380623ef65396efd56"
Nov 02 09:54:23.083043 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:23.083014    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"
Nov 02 09:54:23.083181 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:23.083054    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl" event={"ID":"7e675a17-aed8-48b0-96c9-d9ccfd5c3121","Type":"ContainerDied","Data":"783b73939dac0a9bd4bbe666252d5e4eda1e9a403523f4688710647e3f53d777"}
Nov 02 09:54:23.083181 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:23.083103    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="783b73939dac0a9bd4bbe666252d5e4eda1e9a403523f4688710647e3f53d777"
Nov 02 09:54:23.211875 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:23.211829    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6692fe0c-a649-4c5f-aeb6-bd80777cd4c4" path="/var/lib/kubelet/pods/6692fe0c-a649-4c5f-aeb6-bd80777cd4c4/volumes"
Nov 02 09:54:23.801901 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:23.801866    4449 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-4a99520c97db: secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:23.801901 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:23.801893    4449 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7: secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:23.802109 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:23.801951    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials podName:f1af3d18-a356-48bd-9b1d-ccd78a76e4e0 nodeName:}" failed. No retries permitted until 2026-11-02 09:54:31.801937999 +0000 UTC m=+1148.740224776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7" (UID: "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0") : secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:24.038010 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:24.037976    4449 scope.go:122] "RemoveContainer" containerID="96d5f93cab8251bbbf47aef9563102abc1a3a661e81b834f5b921bbe6bd79307"
Nov 02 09:54:24.052880 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:24.052851    4449 scope.go:122] "RemoveContainer" containerID="32c1498603deeafe49fd9c6b473b8da22b8babc3e505bc57a02a9b0fd2f49b3c"
Nov 02 09:54:24.066622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:24.066600    4449 scope.go:122] "RemoveContainer" containerID="5dfedac38e23f8b0aabfde50038331ae4e45261acfbd40b82a704f2ce2144ac0"
Nov 02 09:54:24.078539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:24.078513    4449 scope.go:122] "RemoveContainer" containerID="64822bafa9166e63c90286251132b14ce08c46133a581f42f8f3f236f1f1f8af"
Nov 02 09:54:24.090436 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:24.090415    4449 scope.go:122] "RemoveContainer" containerID="3c6d78bda774cdc613412a3f52f9d3d7973758c842f1ef87bb14eb04d06b4e14"
Nov 02 09:54:24.105094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:24.105075    4449 scope.go:122] "RemoveContainer" containerID="dfe631ec74ef3a6f9100b9eb17b1ef7678141315e47f65d4909a7752377ce239"
Nov 02 09:54:24.116401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:24.116378    4449 scope.go:122] "RemoveContainer" containerID="49650b5045b4d70e0c6e80f46592299b63371eb3dbd2e626fe748bc4d7b0f86e"
Nov 02 09:54:24.127368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:24.127342    4449 scope.go:122] "RemoveContainer" containerID="0537ea13a742ef03a0f05a79faf2a4a0eb076ea2f87d30372cd623eafd769fd4"
Nov 02 09:54:26.602400 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:26.602360    4449 patch_prober.go:29] interesting pod/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:54:26.602882 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:26.602450    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" podUID="e1934300-75bb-44de-b982-25d0fd418c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:26.696283 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:26.685077    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"]
Nov 02 09:54:26.696283 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:26.695940    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"]
Nov 02 09:54:28.279258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:27.703165    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"]
Nov 02 09:54:28.279258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:27.753620    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:28.279258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:27.883305    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"]
Nov 02 09:54:28.632296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:28.553339    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:28.632296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:28.628676    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"]
Nov 02 09:54:28.638726 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:28.638660    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4d567b-rrm86"]
Nov 02 09:54:29.214638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:29.214605    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda43351-8185-454d-898c-a5b118892ac3" path="/var/lib/kubelet/pods/fda43351-8185-454d-898c-a5b118892ac3/volumes"
Nov 02 09:54:31.030594 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:31.030558    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"]
Nov 02 09:54:31.040751 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:31.040706    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68f86308-h7mvx"]
Nov 02 09:54:31.212706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:31.212675    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67fd243f-cdb4-490e-b83f-41cbbcf209f5" path="/var/lib/kubelet/pods/67fd243f-cdb4-490e-b83f-41cbbcf209f5/volumes"
Nov 02 09:54:31.635321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:31.635192    4449 generic.go:356] "Generic (PLEG): container finished" podID="e1934300-75bb-44de-b982-25d0fd418c46" containerID="fa9782e034b757ed2bbeba83032cea1677860b9c47103939f822b8f660fa9fbf" exitCode=0
Nov 02 09:54:31.635321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:31.635258    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" event={"ID":"e1934300-75bb-44de-b982-25d0fd418c46","Type":"ContainerDied","Data":"fa9782e034b757ed2bbeba83032cea1677860b9c47103939f822b8f660fa9fbf"}
Nov 02 09:54:31.635832 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:31.635798    4449 scope.go:122] "RemoveContainer" containerID="fa9782e034b757ed2bbeba83032cea1677860b9c47103939f822b8f660fa9fbf"
Nov 02 09:54:31.638485 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:31.638450    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:31.638623 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:31.638529    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:31.638623 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:31.638573    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:31.686716 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:31.686518    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2" podUID="b4e13bb8-dd69-470b-b395-769e72deb289" containerName="payload-processing" containerID="cri-o://17285b3dfa55b15d3fc46261a69588bbd2e9e038423b50e6a1f77afd1b99ef5e" gracePeriod=25
Nov 02 09:54:31.696558 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:31.696511    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl" podUID="5ae067e1-9dd1-4445-bd69-4c19f184559d" containerName="payload-pre-processing" containerID="cri-o://11a611ab5ff08d5b2acc171a24bfa1e1b07ce35c485f8473ba0687f9deae0bc4" gracePeriod=25
Nov 02 09:54:31.867829 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:31.865885    4449 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-1521404c1537: secret "agc-cred-1521404c1537" not found
Nov 02 09:54:31.867829 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:31.865917    4449 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-4a99520c97db: secret "agc-cred-4a99520c97db" not found
Nov 02 09:54:31.867829 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:31.865932    4449 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7: [secret "agc-cred-1521404c1537" not found, secret "agc-cred-4a99520c97db" not found]
Nov 02 09:54:31.867829 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:31.865995    4449 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials podName:f1af3d18-a356-48bd-9b1d-ccd78a76e4e0 nodeName:}" failed. No retries permitted until 2026-11-02 09:54:47.865975522 +0000 UTC m=+1164.804262276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7" (UID: "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0") : [secret "agc-cred-1521404c1537" not found, secret "agc-cred-4a99520c97db" not found]
Nov 02 09:54:32.021662 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.016825    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"
Nov 02 09:54:32.070385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.069980    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"
Nov 02 09:54:32.171707 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.171670    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e13bb8-dd69-470b-b395-769e72deb289-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e13bb8-dd69-470b-b395-769e72deb289-plugins-config-volume\") pod \"b4e13bb8-dd69-470b-b395-769e72deb289\" (UID: \"b4e13bb8-dd69-470b-b395-769e72deb289\") "
Nov 02 09:54:32.171895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.171738    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae067e1-9dd1-4445-bd69-4c19f184559d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae067e1-9dd1-4445-bd69-4c19f184559d-plugins-config-volume\") pod \"5ae067e1-9dd1-4445-bd69-4c19f184559d\" (UID: \"5ae067e1-9dd1-4445-bd69-4c19f184559d\") "
Nov 02 09:54:32.171895 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.171835    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e13bb8-dd69-470b-b395-769e72deb289-kube-api-access-b5s2v\" (UniqueName: \"kubernetes.io/projected/b4e13bb8-dd69-470b-b395-769e72deb289-kube-api-access-b5s2v\") pod \"b4e13bb8-dd69-470b-b395-769e72deb289\" (UID: \"b4e13bb8-dd69-470b-b395-769e72deb289\") "
Nov 02 09:54:32.172024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.171895    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae067e1-9dd1-4445-bd69-4c19f184559d-kube-api-access-8xlp4\" (UniqueName: \"kubernetes.io/projected/5ae067e1-9dd1-4445-bd69-4c19f184559d-kube-api-access-8xlp4\") pod \"5ae067e1-9dd1-4445-bd69-4c19f184559d\" (UID: \"5ae067e1-9dd1-4445-bd69-4c19f184559d\") "
Nov 02 09:54:32.172116 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.172082    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e13bb8-dd69-470b-b395-769e72deb289-plugins-config-volume" pod "b4e13bb8-dd69-470b-b395-769e72deb289" (UID: "b4e13bb8-dd69-470b-b395-769e72deb289"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:32.172184 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.172136    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae067e1-9dd1-4445-bd69-4c19f184559d-plugins-config-volume" pod "5ae067e1-9dd1-4445-bd69-4c19f184559d" (UID: "5ae067e1-9dd1-4445-bd69-4c19f184559d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:32.172868 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.172825    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e13bb8-dd69-470b-b395-769e72deb289-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:32.172868 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.172856    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae067e1-9dd1-4445-bd69-4c19f184559d-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:32.174871 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.174840    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e13bb8-dd69-470b-b395-769e72deb289-kube-api-access-b5s2v" pod "b4e13bb8-dd69-470b-b395-769e72deb289" (UID: "b4e13bb8-dd69-470b-b395-769e72deb289"). InnerVolumeSpecName "kube-api-access-b5s2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:32.175266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.175229    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae067e1-9dd1-4445-bd69-4c19f184559d-kube-api-access-8xlp4" pod "5ae067e1-9dd1-4445-bd69-4c19f184559d" (UID: "5ae067e1-9dd1-4445-bd69-4c19f184559d"). InnerVolumeSpecName "kube-api-access-8xlp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:32.274331 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.274210    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5s2v\" (UniqueName: \"kubernetes.io/projected/b4e13bb8-dd69-470b-b395-769e72deb289-kube-api-access-b5s2v\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:32.274331 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.274276    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xlp4\" (UniqueName: \"kubernetes.io/projected/5ae067e1-9dd1-4445-bd69-4c19f184559d-kube-api-access-8xlp4\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:32.430680 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.430647    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:32.430767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.430714    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:32.698971 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.698925    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" event={"ID":"e1934300-75bb-44de-b982-25d0fd418c46","Type":"ContainerStarted","Data":"d503d528b4c2cb7315cdfc401effb7ae267f787fc587e30285c6d3966430934b"}
Nov 02 09:54:32.704027 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.703839    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw" podUID="07d6f736-47e3-4f7f-b866-aea262f8fc3b" containerName="payload-processing" containerID="cri-o://d20ea2220c915388a746d68a672cb01b73b9c52d40a69b62fe38e0df00448d23" gracePeriod=25
Nov 02 09:54:32.710962 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.710937    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch" event={"ID":"d1af78de-9573-422d-8563-4a32acc6e833","Type":"ContainerStarted","Data":"e83bdb7b39407e4b99587cc4e2cdf125e26e9dd49f23a0542ea1c776a6c50ba3"}
Nov 02 09:54:32.711831 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.711814    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:32.713380 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.713364    4449 generic.go:356] "Generic (PLEG): container finished" podID="b4e13bb8-dd69-470b-b395-769e72deb289" containerID="17285b3dfa55b15d3fc46261a69588bbd2e9e038423b50e6a1f77afd1b99ef5e" exitCode=0
Nov 02 09:54:32.713557 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.713539    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"
Nov 02 09:54:32.713632 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.713499    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2" event={"ID":"b4e13bb8-dd69-470b-b395-769e72deb289","Type":"ContainerDied","Data":"17285b3dfa55b15d3fc46261a69588bbd2e9e038423b50e6a1f77afd1b99ef5e"}
Nov 02 09:54:32.713723 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.713709    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2" event={"ID":"b4e13bb8-dd69-470b-b395-769e72deb289","Type":"ContainerDied","Data":"3a265c0fe0d0a4dad55b1e9de72c037172c74300278bbe2976decc7249497cd4"}
Nov 02 09:54:32.713805 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.713730    4449 scope.go:122] "RemoveContainer" containerID="17285b3dfa55b15d3fc46261a69588bbd2e9e038423b50e6a1f77afd1b99ef5e"
Nov 02 09:54:32.717535 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.717514    4449 generic.go:356] "Generic (PLEG): container finished" podID="5ae067e1-9dd1-4445-bd69-4c19f184559d" containerID="11a611ab5ff08d5b2acc171a24bfa1e1b07ce35c485f8473ba0687f9deae0bc4" exitCode=0
Nov 02 09:54:32.717623 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.717604    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl" event={"ID":"5ae067e1-9dd1-4445-bd69-4c19f184559d","Type":"ContainerDied","Data":"11a611ab5ff08d5b2acc171a24bfa1e1b07ce35c485f8473ba0687f9deae0bc4"}
Nov 02 09:54:32.717664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.717647    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl" event={"ID":"5ae067e1-9dd1-4445-bd69-4c19f184559d","Type":"ContainerDied","Data":"0a3840180d32d5779b73676cd05571b48f269bcf11e709146824538aff91878b"}
Nov 02 09:54:32.717745 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.717730    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"
Nov 02 09:54:32.730739 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.730719    4449 scope.go:122] "RemoveContainer" containerID="17285b3dfa55b15d3fc46261a69588bbd2e9e038423b50e6a1f77afd1b99ef5e"
Nov 02 09:54:32.731038 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:32.731017    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17285b3dfa55b15d3fc46261a69588bbd2e9e038423b50e6a1f77afd1b99ef5e\": container with ID starting with 17285b3dfa55b15d3fc46261a69588bbd2e9e038423b50e6a1f77afd1b99ef5e not found: ID does not exist" containerID="17285b3dfa55b15d3fc46261a69588bbd2e9e038423b50e6a1f77afd1b99ef5e"
Nov 02 09:54:32.731102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.731047    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17285b3dfa55b15d3fc46261a69588bbd2e9e038423b50e6a1f77afd1b99ef5e"} err="failed to get container status \"17285b3dfa55b15d3fc46261a69588bbd2e9e038423b50e6a1f77afd1b99ef5e\": rpc error: code = NotFound desc = could not find container \"17285b3dfa55b15d3fc46261a69588bbd2e9e038423b50e6a1f77afd1b99ef5e\": container with ID starting with 17285b3dfa55b15d3fc46261a69588bbd2e9e038423b50e6a1f77afd1b99ef5e not found: ID does not exist"
Nov 02 09:54:32.731102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.731065    4449 scope.go:122] "RemoveContainer" containerID="11a611ab5ff08d5b2acc171a24bfa1e1b07ce35c485f8473ba0687f9deae0bc4"
Nov 02 09:54:32.783152 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.783062    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch" podStartSLOduration=4.252545856 podStartE2EDuration="16.783046893s" totalImagesPullingTime="12.530501037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:32.748750254 +0000 UTC m=+1149.687037055" watchObservedRunningTime="2026-11-02 09:54:32.783046893 +0000 UTC m=+1149.721333669"
Nov 02 09:54:32.820009 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.819932    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"]
Nov 02 09:54:32.824768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.824748    4449 scope.go:122] "RemoveContainer" containerID="11a611ab5ff08d5b2acc171a24bfa1e1b07ce35c485f8473ba0687f9deae0bc4"
Nov 02 09:54:32.825126 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:32.825105    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a611ab5ff08d5b2acc171a24bfa1e1b07ce35c485f8473ba0687f9deae0bc4\": container with ID starting with 11a611ab5ff08d5b2acc171a24bfa1e1b07ce35c485f8473ba0687f9deae0bc4 not found: ID does not exist" containerID="11a611ab5ff08d5b2acc171a24bfa1e1b07ce35c485f8473ba0687f9deae0bc4"
Nov 02 09:54:32.825210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.825135    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a611ab5ff08d5b2acc171a24bfa1e1b07ce35c485f8473ba0687f9deae0bc4"} err="failed to get container status \"11a611ab5ff08d5b2acc171a24bfa1e1b07ce35c485f8473ba0687f9deae0bc4\": rpc error: code = NotFound desc = could not find container \"11a611ab5ff08d5b2acc171a24bfa1e1b07ce35c485f8473ba0687f9deae0bc4\": container with ID starting with 11a611ab5ff08d5b2acc171a24bfa1e1b07ce35c485f8473ba0687f9deae0bc4 not found: ID does not exist"
Nov 02 09:54:32.827550 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.827530    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-13c6a8db-646b88c58c-69lf2"]
Nov 02 09:54:32.838947 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.838921    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"]
Nov 02 09:54:32.848346 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.848323    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-13c6a8db-c677584cf-6h9fl"]
Nov 02 09:54:32.885142 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:32.885093    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7" podUID="f1af3d18-a356-48bd-9b1d-ccd78a76e4e0" containerName="payload-processing" containerID="cri-o://d24e78933907ddae3fdb3aff67fb8b3c2b7adfa1964826ca6ae7ae289e9b1326" gracePeriod=25
Nov 02 09:54:33.136467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.136428    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:33.217791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.214122    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-provider-credentials\") pod \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") "
Nov 02 09:54:33.217791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.214185    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-plugins-config-volume\") pod \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") "
Nov 02 09:54:33.217791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.214235    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-kube-api-access-jn5dp\" (UniqueName: \"kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-kube-api-access-jn5dp\") pod \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") "
Nov 02 09:54:33.217791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.214498    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-routing-overlay-volume\") pod \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\" (UID: \"07d6f736-47e3-4f7f-b866-aea262f8fc3b\") "
Nov 02 09:54:33.217791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.215146    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-plugins-config-volume" pod "07d6f736-47e3-4f7f-b866-aea262f8fc3b" (UID: "07d6f736-47e3-4f7f-b866-aea262f8fc3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:33.217791 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.215493    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-routing-overlay-volume" pod "07d6f736-47e3-4f7f-b866-aea262f8fc3b" (UID: "07d6f736-47e3-4f7f-b866-aea262f8fc3b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:33.227768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.221651    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-kube-api-access-jn5dp" pod "07d6f736-47e3-4f7f-b866-aea262f8fc3b" (UID: "07d6f736-47e3-4f7f-b866-aea262f8fc3b"). InnerVolumeSpecName "kube-api-access-jn5dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:33.227768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.225113    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae067e1-9dd1-4445-bd69-4c19f184559d" path="/var/lib/kubelet/pods/5ae067e1-9dd1-4445-bd69-4c19f184559d/volumes"
Nov 02 09:54:33.227768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.225767    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e13bb8-dd69-470b-b395-769e72deb289" path="/var/lib/kubelet/pods/b4e13bb8-dd69-470b-b395-769e72deb289/volumes"
Nov 02 09:54:33.234298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.234264    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-provider-credentials" pod "07d6f736-47e3-4f7f-b866-aea262f8fc3b" (UID: "07d6f736-47e3-4f7f-b866-aea262f8fc3b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:33.317301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.317162    4449 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-routing-overlay-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:33.317543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.317524    4449 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-provider-credentials\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:33.317640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.317627    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d6f736-47e3-4f7f-b866-aea262f8fc3b-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:33.317715 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.317705    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn5dp\" (UniqueName: \"kubernetes.io/projected/07d6f736-47e3-4f7f-b866-aea262f8fc3b-kube-api-access-jn5dp\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:33.421188 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.421153    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:33.521951 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.521906    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-kube-api-access-48zth\" (UniqueName: \"kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-kube-api-access-48zth\") pod \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") "
Nov 02 09:54:33.522176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.521990    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-plugins-config-volume\") pod \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") "
Nov 02 09:54:33.522176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.522056    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-routing-overlay-volume\") pod \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") "
Nov 02 09:54:33.522346 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.522195    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials\") pod \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\" (UID: \"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0\") "
Nov 02 09:54:33.524533 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.524418    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-routing-overlay-volume" pod "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0" (UID: "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:33.524674 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.524655    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-plugins-config-volume" pod "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0" (UID: "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:33.527598 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.527564    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials" pod "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0" (UID: "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:33.540423 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.540381    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-kube-api-access-48zth" pod "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0" (UID: "f1af3d18-a356-48bd-9b1d-ccd78a76e4e0"). InnerVolumeSpecName "kube-api-access-48zth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:33.540600 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.540448    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:54:33.543260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.542920    4449 patch_prober.go:29] interesting pod/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 02 09:54:33.543260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.542981    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" podUID="e1934300-75bb-44de-b982-25d0fd418c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 02 09:54:33.625673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.624723    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48zth\" (UniqueName: \"kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-kube-api-access-48zth\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:33.625673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.624758    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:33.625673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.624775    4449 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-routing-overlay-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:33.625673 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.624790    4449 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0-provider-credentials\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:33.728231 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.728125    4449 generic.go:356] "Generic (PLEG): container finished" podID="07d6f736-47e3-4f7f-b866-aea262f8fc3b" containerID="d20ea2220c915388a746d68a672cb01b73b9c52d40a69b62fe38e0df00448d23" exitCode=0
Nov 02 09:54:33.728231 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.728179    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"
Nov 02 09:54:33.728474 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.728202    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw" event={"ID":"07d6f736-47e3-4f7f-b866-aea262f8fc3b","Type":"ContainerDied","Data":"d20ea2220c915388a746d68a672cb01b73b9c52d40a69b62fe38e0df00448d23"}
Nov 02 09:54:33.728474 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.728292    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw" event={"ID":"07d6f736-47e3-4f7f-b866-aea262f8fc3b","Type":"ContainerDied","Data":"cd539d55869dbcc617ada06e85cd8a36d539912b88a564ccf3f29352b39c9c47"}
Nov 02 09:54:33.728474 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.728323    4449 scope.go:122] "RemoveContainer" containerID="d20ea2220c915388a746d68a672cb01b73b9c52d40a69b62fe38e0df00448d23"
Nov 02 09:54:33.729955 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.729931    4449 generic.go:356] "Generic (PLEG): container finished" podID="f1af3d18-a356-48bd-9b1d-ccd78a76e4e0" containerID="d24e78933907ddae3fdb3aff67fb8b3c2b7adfa1964826ca6ae7ae289e9b1326" exitCode=0
Nov 02 09:54:33.730050 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.729966    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7" event={"ID":"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0","Type":"ContainerDied","Data":"d24e78933907ddae3fdb3aff67fb8b3c2b7adfa1964826ca6ae7ae289e9b1326"}
Nov 02 09:54:33.730050 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.730017    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7" event={"ID":"f1af3d18-a356-48bd-9b1d-ccd78a76e4e0","Type":"ContainerDied","Data":"570465c8eaba0ff7ac1cb916103ce98a5d102981f52f2c97b0ff40643bec6401"}
Nov 02 09:54:33.730050 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.729979    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"
Nov 02 09:54:33.741360 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.741282    4449 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-scwch container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 09:54:33.741453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.741367    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch" podUID="d1af78de-9573-422d-8563-4a32acc6e833" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 09:54:33.743131 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.743115    4449 scope.go:122] "RemoveContainer" containerID="d20ea2220c915388a746d68a672cb01b73b9c52d40a69b62fe38e0df00448d23"
Nov 02 09:54:33.743412 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:33.743394    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20ea2220c915388a746d68a672cb01b73b9c52d40a69b62fe38e0df00448d23\": container with ID starting with d20ea2220c915388a746d68a672cb01b73b9c52d40a69b62fe38e0df00448d23 not found: ID does not exist" containerID="d20ea2220c915388a746d68a672cb01b73b9c52d40a69b62fe38e0df00448d23"
Nov 02 09:54:33.743474 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.743418    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20ea2220c915388a746d68a672cb01b73b9c52d40a69b62fe38e0df00448d23"} err="failed to get container status \"d20ea2220c915388a746d68a672cb01b73b9c52d40a69b62fe38e0df00448d23\": rpc error: code = NotFound desc = could not find container \"d20ea2220c915388a746d68a672cb01b73b9c52d40a69b62fe38e0df00448d23\": container with ID starting with d20ea2220c915388a746d68a672cb01b73b9c52d40a69b62fe38e0df00448d23 not found: ID does not exist"
Nov 02 09:54:33.743474 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.743435    4449 scope.go:122] "RemoveContainer" containerID="d24e78933907ddae3fdb3aff67fb8b3c2b7adfa1964826ca6ae7ae289e9b1326"
Nov 02 09:54:33.782303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.782281    4449 scope.go:122] "RemoveContainer" containerID="d24e78933907ddae3fdb3aff67fb8b3c2b7adfa1964826ca6ae7ae289e9b1326"
Nov 02 09:54:33.782666 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:33.782647    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24e78933907ddae3fdb3aff67fb8b3c2b7adfa1964826ca6ae7ae289e9b1326\": container with ID starting with d24e78933907ddae3fdb3aff67fb8b3c2b7adfa1964826ca6ae7ae289e9b1326 not found: ID does not exist" containerID="d24e78933907ddae3fdb3aff67fb8b3c2b7adfa1964826ca6ae7ae289e9b1326"
Nov 02 09:54:33.782730 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.782679    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24e78933907ddae3fdb3aff67fb8b3c2b7adfa1964826ca6ae7ae289e9b1326"} err="failed to get container status \"d24e78933907ddae3fdb3aff67fb8b3c2b7adfa1964826ca6ae7ae289e9b1326\": rpc error: code = NotFound desc = could not find container \"d24e78933907ddae3fdb3aff67fb8b3c2b7adfa1964826ca6ae7ae289e9b1326\": container with ID starting with d24e78933907ddae3fdb3aff67fb8b3c2b7adfa1964826ca6ae7ae289e9b1326 not found: ID does not exist"
Nov 02 09:54:33.784833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.784783    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"]
Nov 02 09:54:33.791103 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.791073    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8598ddsbw"]
Nov 02 09:54:33.797163 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.797140    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"]
Nov 02 09:54:33.802519 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:33.802495    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898wl7l7"]
Nov 02 09:54:34.540657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:34.540623    4449 patch_prober.go:29] interesting pod/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 02 09:54:34.541065 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:34.540682    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" podUID="e1934300-75bb-44de-b982-25d0fd418c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 02 09:54:34.772250 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:34.770472    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 09:54:35.208382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:35.208344    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d6f736-47e3-4f7f-b866-aea262f8fc3b" path="/var/lib/kubelet/pods/07d6f736-47e3-4f7f-b866-aea262f8fc3b/volumes"
Nov 02 09:54:35.208818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:35.208804    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1af3d18-a356-48bd-9b1d-ccd78a76e4e0" path="/var/lib/kubelet/pods/f1af3d18-a356-48bd-9b1d-ccd78a76e4e0/volumes"
Nov 02 09:54:35.540491 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:35.540454    4449 patch_prober.go:29] interesting pod/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 02 09:54:35.540640 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:35.540518    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" podUID="e1934300-75bb-44de-b982-25d0fd418c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 02 09:54:36.540888 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:36.540627    4449 patch_prober.go:29] interesting pod/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 02 09:54:36.540888 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:36.540681    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" podUID="e1934300-75bb-44de-b982-25d0fd418c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 02 09:54:36.559245 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:36.549460    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:54:37.302278 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:37.300197    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"]
Nov 02 09:54:37.540459 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:37.540427    4449 patch_prober.go:29] interesting pod/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 02 09:54:37.540645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:37.540501    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" podUID="e1934300-75bb-44de-b982-25d0fd418c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 02 09:54:37.605422 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:37.605308    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"]
Nov 02 09:54:37.613641 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:37.613588    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-13c6a8db-g6dzw"]
Nov 02 09:54:37.744277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:37.740664    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"]
Nov 02 09:54:37.744277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:37.740935    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" podUID="e1934300-75bb-44de-b982-25d0fd418c46" containerName="istio-proxy" containerID="cri-o://d503d528b4c2cb7315cdfc401effb7ae267f787fc587e30285c6d3966430934b" gracePeriod=30
Nov 02 09:54:39.302535 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:39.302481    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f619613-433a-4f61-827b-9fdc47641b23" path="/var/lib/kubelet/pods/3f619613-433a-4f61-827b-9fdc47641b23/volumes"
Nov 02 09:54:40.034225 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:40.034187    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"]
Nov 02 09:54:40.044396 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:40.044306    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc58f74-2s5hz"]
Nov 02 09:54:40.550147 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:40.550101    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:54:41.251253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:41.245822    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f5b50ce-7305-4667-bb40-0b3cad698096" path="/var/lib/kubelet/pods/5f5b50ce-7305-4667-bb40-0b3cad698096/volumes"
Nov 02 09:54:43.702726 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.702698    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:54:43.752800 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.752769    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1934300-75bb-44de-b982-25d0fd418c46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1934300-75bb-44de-b982-25d0fd418c46-istio-podinfo\") pod \"e1934300-75bb-44de-b982-25d0fd418c46\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") "
Nov 02 09:54:43.752998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.752829    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-certs\") pod \"e1934300-75bb-44de-b982-25d0fd418c46\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") "
Nov 02 09:54:43.752998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.752852    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-envoy\") pod \"e1934300-75bb-44de-b982-25d0fd418c46\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") "
Nov 02 09:54:43.752998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.752907    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-kube-api-access-v2mdl\" (UniqueName: \"kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-kube-api-access-v2mdl\") pod \"e1934300-75bb-44de-b982-25d0fd418c46\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") "
Nov 02 09:54:43.753188 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.753022    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-data\") pod \"e1934300-75bb-44de-b982-25d0fd418c46\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") "
Nov 02 09:54:43.753188 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.753096    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-socket\") pod \"e1934300-75bb-44de-b982-25d0fd418c46\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") "
Nov 02 09:54:43.753332 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.753205    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-credential-socket\") pod \"e1934300-75bb-44de-b982-25d0fd418c46\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") "
Nov 02 09:54:43.753332 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.753304    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1934300-75bb-44de-b982-25d0fd418c46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1934300-75bb-44de-b982-25d0fd418c46-istiod-ca-cert\") pod \"e1934300-75bb-44de-b982-25d0fd418c46\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") "
Nov 02 09:54:43.753332 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.753329    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-istio-token\" (UniqueName: \"kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-istio-token\") pod \"e1934300-75bb-44de-b982-25d0fd418c46\" (UID: \"e1934300-75bb-44de-b982-25d0fd418c46\") "
Nov 02 09:54:43.753586 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.753553    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-certs" pod "e1934300-75bb-44de-b982-25d0fd418c46" (UID: "e1934300-75bb-44de-b982-25d0fd418c46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:54:43.753658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.753571    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-credential-socket" pod "e1934300-75bb-44de-b982-25d0fd418c46" (UID: "e1934300-75bb-44de-b982-25d0fd418c46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:54:43.753658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.753578    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-socket" pod "e1934300-75bb-44de-b982-25d0fd418c46" (UID: "e1934300-75bb-44de-b982-25d0fd418c46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:54:43.753658 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.753636    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-data" pod "e1934300-75bb-44de-b982-25d0fd418c46" (UID: "e1934300-75bb-44de-b982-25d0fd418c46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:54:43.753943 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.753920    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1934300-75bb-44de-b982-25d0fd418c46-istiod-ca-cert" pod "e1934300-75bb-44de-b982-25d0fd418c46" (UID: "e1934300-75bb-44de-b982-25d0fd418c46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:43.754116 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.754091    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:43.754198 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.754117    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:43.754198 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.754132    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:43.754198 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.754142    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:43.754198 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.754150    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1934300-75bb-44de-b982-25d0fd418c46-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:43.757936 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.757907    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-kube-api-access-v2mdl" pod "e1934300-75bb-44de-b982-25d0fd418c46" (UID: "e1934300-75bb-44de-b982-25d0fd418c46"). InnerVolumeSpecName "kube-api-access-v2mdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:43.758079 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.757950    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-istio-token" pod "e1934300-75bb-44de-b982-25d0fd418c46" (UID: "e1934300-75bb-44de-b982-25d0fd418c46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:43.758079 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.757989    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1934300-75bb-44de-b982-25d0fd418c46-istio-podinfo" pod "e1934300-75bb-44de-b982-25d0fd418c46" (UID: "e1934300-75bb-44de-b982-25d0fd418c46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:54:43.758375 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.758353    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-envoy" pod "e1934300-75bb-44de-b982-25d0fd418c46" (UID: "e1934300-75bb-44de-b982-25d0fd418c46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:54:43.855323 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.855286    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:43.855323 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.855317    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1934300-75bb-44de-b982-25d0fd418c46-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:43.855539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.855332    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1934300-75bb-44de-b982-25d0fd418c46-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:43.855539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.855346    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2mdl\" (UniqueName: \"kubernetes.io/projected/e1934300-75bb-44de-b982-25d0fd418c46-kube-api-access-v2mdl\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:43.899537 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.899503    4449 generic.go:356] "Generic (PLEG): container finished" podID="e1934300-75bb-44de-b982-25d0fd418c46" containerID="d503d528b4c2cb7315cdfc401effb7ae267f787fc587e30285c6d3966430934b" exitCode=0
Nov 02 09:54:43.899701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.899626    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"
Nov 02 09:54:43.899701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.899685    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" event={"ID":"e1934300-75bb-44de-b982-25d0fd418c46","Type":"ContainerDied","Data":"d503d528b4c2cb7315cdfc401effb7ae267f787fc587e30285c6d3966430934b"}
Nov 02 09:54:43.899834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.899736    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw" event={"ID":"e1934300-75bb-44de-b982-25d0fd418c46","Type":"ContainerDied","Data":"5641c8300e9214f729016bfcf3bc77ba130648a271b22de005d902ab9f5d0921"}
Nov 02 09:54:43.899834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.899766    4449 scope.go:122] "RemoveContainer" containerID="d503d528b4c2cb7315cdfc401effb7ae267f787fc587e30285c6d3966430934b"
Nov 02 09:54:43.916195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.916174    4449 scope.go:122] "RemoveContainer" containerID="fa9782e034b757ed2bbeba83032cea1677860b9c47103939f822b8f660fa9fbf"
Nov 02 09:54:43.928120 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.928098    4449 scope.go:122] "RemoveContainer" containerID="d503d528b4c2cb7315cdfc401effb7ae267f787fc587e30285c6d3966430934b"
Nov 02 09:54:43.928494 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:43.928455    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d503d528b4c2cb7315cdfc401effb7ae267f787fc587e30285c6d3966430934b\": container with ID starting with d503d528b4c2cb7315cdfc401effb7ae267f787fc587e30285c6d3966430934b not found: ID does not exist" containerID="d503d528b4c2cb7315cdfc401effb7ae267f787fc587e30285c6d3966430934b"
Nov 02 09:54:43.928682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.928493    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d503d528b4c2cb7315cdfc401effb7ae267f787fc587e30285c6d3966430934b"} err="failed to get container status \"d503d528b4c2cb7315cdfc401effb7ae267f787fc587e30285c6d3966430934b\": rpc error: code = NotFound desc = could not find container \"d503d528b4c2cb7315cdfc401effb7ae267f787fc587e30285c6d3966430934b\": container with ID starting with d503d528b4c2cb7315cdfc401effb7ae267f787fc587e30285c6d3966430934b not found: ID does not exist"
Nov 02 09:54:43.928682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.928540    4449 scope.go:122] "RemoveContainer" containerID="fa9782e034b757ed2bbeba83032cea1677860b9c47103939f822b8f660fa9fbf"
Nov 02 09:54:43.929472 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:43.929445    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9782e034b757ed2bbeba83032cea1677860b9c47103939f822b8f660fa9fbf\": container with ID starting with fa9782e034b757ed2bbeba83032cea1677860b9c47103939f822b8f660fa9fbf not found: ID does not exist" containerID="fa9782e034b757ed2bbeba83032cea1677860b9c47103939f822b8f660fa9fbf"
Nov 02 09:54:43.929560 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.929480    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9782e034b757ed2bbeba83032cea1677860b9c47103939f822b8f660fa9fbf"} err="failed to get container status \"fa9782e034b757ed2bbeba83032cea1677860b9c47103939f822b8f660fa9fbf\": rpc error: code = NotFound desc = could not find container \"fa9782e034b757ed2bbeba83032cea1677860b9c47103939f822b8f660fa9fbf\": container with ID starting with fa9782e034b757ed2bbeba83032cea1677860b9c47103939f822b8f660fa9fbf not found: ID does not exist"
Nov 02 09:54:43.933620 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.933596    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"]
Nov 02 09:54:43.939550 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:43.939526    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-13c6a8db-openshift-default-797684c4c7-7b6lw"]
Nov 02 09:54:44.066069 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066040    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"]
Nov 02 09:54:44.066731 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066716    4449 cpu_manager.go:414] "containerMap: removing container" podUID="07d6f736-47e3-4f7f-b866-aea262f8fc3b" containerName="payload-processing"
Nov 02 09:54:44.066782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066733    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d6f736-47e3-4f7f-b866-aea262f8fc3b" containerName="payload-processing"
Nov 02 09:54:44.066782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066747    4449 cpu_manager.go:414] "containerMap: removing container" podUID="e1934300-75bb-44de-b982-25d0fd418c46" containerName="istio-proxy"
Nov 02 09:54:44.066782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066752    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1934300-75bb-44de-b982-25d0fd418c46" containerName="istio-proxy"
Nov 02 09:54:44.066782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066764    4449 cpu_manager.go:414] "containerMap: removing container" podUID="7e675a17-aed8-48b0-96c9-d9ccfd5c3121" containerName="revoke-keys"
Nov 02 09:54:44.066782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066769    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e675a17-aed8-48b0-96c9-d9ccfd5c3121" containerName="revoke-keys"
Nov 02 09:54:44.066928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066786    4449 cpu_manager.go:414] "containerMap: removing container" podUID="e1934300-75bb-44de-b982-25d0fd418c46" containerName="istio-proxy"
Nov 02 09:54:44.066928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066791    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1934300-75bb-44de-b982-25d0fd418c46" containerName="istio-proxy"
Nov 02 09:54:44.066928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066806    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f1af3d18-a356-48bd-9b1d-ccd78a76e4e0" containerName="payload-processing"
Nov 02 09:54:44.066928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066811    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1af3d18-a356-48bd-9b1d-ccd78a76e4e0" containerName="payload-processing"
Nov 02 09:54:44.066928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066833    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5ae067e1-9dd1-4445-bd69-4c19f184559d" containerName="payload-pre-processing"
Nov 02 09:54:44.066928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066837    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae067e1-9dd1-4445-bd69-4c19f184559d" containerName="payload-pre-processing"
Nov 02 09:54:44.066928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066846    4449 cpu_manager.go:414] "containerMap: removing container" podUID="6692fe0c-a649-4c5f-aeb6-bd80777cd4c4" containerName="payload-processing"
Nov 02 09:54:44.066928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066851    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6692fe0c-a649-4c5f-aeb6-bd80777cd4c4" containerName="payload-processing"
Nov 02 09:54:44.066928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066859    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b4e13bb8-dd69-470b-b395-769e72deb289" containerName="payload-processing"
Nov 02 09:54:44.066928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066863    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e13bb8-dd69-470b-b395-769e72deb289" containerName="payload-processing"
Nov 02 09:54:44.066928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066872    4449 cpu_manager.go:414] "containerMap: removing container" podUID="3f619613-433a-4f61-827b-9fdc47641b23" containerName="revoke-keys"
Nov 02 09:54:44.066928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066876    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f619613-433a-4f61-827b-9fdc47641b23" containerName="revoke-keys"
Nov 02 09:54:44.067316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066973    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1af3d18-a356-48bd-9b1d-ccd78a76e4e0" containerName="payload-processing"
Nov 02 09:54:44.067316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066984    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae067e1-9dd1-4445-bd69-4c19f184559d" containerName="payload-pre-processing"
Nov 02 09:54:44.067316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066992    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="6692fe0c-a649-4c5f-aeb6-bd80777cd4c4" containerName="payload-processing"
Nov 02 09:54:44.067316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.066999    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e13bb8-dd69-470b-b395-769e72deb289" containerName="payload-processing"
Nov 02 09:54:44.067316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.067005    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f619613-433a-4f61-827b-9fdc47641b23" containerName="revoke-keys"
Nov 02 09:54:44.067316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.067013    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d6f736-47e3-4f7f-b866-aea262f8fc3b" containerName="payload-processing"
Nov 02 09:54:44.067316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.067018    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1934300-75bb-44de-b982-25d0fd418c46" containerName="istio-proxy"
Nov 02 09:54:44.067316 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.067023    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e675a17-aed8-48b0-96c9-d9ccfd5c3121" containerName="revoke-keys"
Nov 02 09:54:44.072210 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.072190    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.078145 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.077953    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"e2e-cmp-legacy-332b1b-provider-cred\""
Nov 02 09:54:44.078145 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.078008    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"routing-overlay\""
Nov 02 09:54:44.080793 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.080636    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"kube-root-ca.crt\""
Nov 02 09:54:44.086011 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.085991    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"]
Nov 02 09:54:44.189358 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.188650    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-e2e--7be3f336-dockercfg-9grkv\""
Nov 02 09:54:44.189358 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.188830    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-plugins-e2e-extmod-a-629129\""
Nov 02 09:54:44.189358 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.188941    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"openshift-service-ca.crt\""
Nov 02 09:54:44.267241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.267083    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fwf7\" (UniqueName: \"kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-kube-api-access-7fwf7\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.267453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.267302    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.267520 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.267474    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.271237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.267645    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.368768 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.368732    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.368975 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.368816    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fwf7\" (UniqueName: \"kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-kube-api-access-7fwf7\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.368975 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.368946    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.369097 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.369004    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.369589 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.369561    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.369589 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.369570    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.373274 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.372495    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.382672 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.382638    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fwf7\" (UniqueName: \"kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-kube-api-access-7fwf7\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.388299 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.388276    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.703884 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:54:44.703852    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a2bffe_6868_4ac6_9e24_e59d84441b77.slice/crio-a4c1d077e7637c030019ad6d4c63d7815927750a0b6c113164ca40c1d345e7df WatchSource:0}: Error finding container a4c1d077e7637c030019ad6d4c63d7815927750a0b6c113164ca40c1d345e7df: Status 404 returned error can't find the container with id a4c1d077e7637c030019ad6d4c63d7815927750a0b6c113164ca40c1d345e7df
Nov 02 09:54:44.706279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.706190    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"]
Nov 02 09:54:44.912290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.912255    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8" event={"ID":"33a2bffe-6868-4ac6-9e24-e59d84441b77","Type":"ContainerStarted","Data":"a3fab60121be25f772a57cbc423666e0230883b7066ac62ddb3d9334b354871a"}
Nov 02 09:54:44.912290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.912304    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8" event={"ID":"33a2bffe-6868-4ac6-9e24-e59d84441b77","Type":"ContainerStarted","Data":"a4c1d077e7637c030019ad6d4c63d7815927750a0b6c113164ca40c1d345e7df"}
Nov 02 09:54:44.912572 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.912355    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:44.935357 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:44.935299    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8" podStartSLOduration=0.935282974 podStartE2EDuration="935.282974ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:44.931672639 +0000 UTC m=+1161.869959434" watchObservedRunningTime="2026-11-02 09:54:44.935282974 +0000 UTC m=+1161.873569747"
Nov 02 09:54:45.211146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.211110    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1934300-75bb-44de-b982-25d0fd418c46" path="/var/lib/kubelet/pods/e1934300-75bb-44de-b982-25d0fd418c46/volumes"
Nov 02 09:54:45.296641 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.296596    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"]
Nov 02 09:54:45.297451 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.297435    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1934300-75bb-44de-b982-25d0fd418c46" containerName="istio-proxy"
Nov 02 09:54:45.301565 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.301547    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"
Nov 02 09:54:45.312175 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.311315    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a71884-dockercfg-7fcqq\""
Nov 02 09:54:45.312175 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.311934    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a71884\""
Nov 02 09:54:45.313876 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.313848    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"]
Nov 02 09:54:45.323835 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.323806    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"]
Nov 02 09:54:45.329838 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.329811    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"
Nov 02 09:54:45.343006 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.342979    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"]
Nov 02 09:54:45.385929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.385891    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqfjk\" (UniqueName: \"kubernetes.io/projected/98dd9543-5592-461c-94e8-7fffe7079d7f-kube-api-access-kqfjk\") pod \"payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf\" (UID: \"98dd9543-5592-461c-94e8-7fffe7079d7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"
Nov 02 09:54:45.386145 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.386098    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq4dk\" (UniqueName: \"kubernetes.io/projected/11a79a51-82c7-4a50-b8aa-6a155be721e3-kube-api-access-tq4dk\") pod \"payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm\" (UID: \"11a79a51-82c7-4a50-b8aa-6a155be721e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"
Nov 02 09:54:45.386236 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.386149    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a79a51-82c7-4a50-b8aa-6a155be721e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm\" (UID: \"11a79a51-82c7-4a50-b8aa-6a155be721e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"
Nov 02 09:54:45.386374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.386357    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98dd9543-5592-461c-94e8-7fffe7079d7f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf\" (UID: \"98dd9543-5592-461c-94e8-7fffe7079d7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"
Nov 02 09:54:45.491295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.491121    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq4dk\" (UniqueName: \"kubernetes.io/projected/11a79a51-82c7-4a50-b8aa-6a155be721e3-kube-api-access-tq4dk\") pod \"payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm\" (UID: \"11a79a51-82c7-4a50-b8aa-6a155be721e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"
Nov 02 09:54:45.491295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.491182    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a79a51-82c7-4a50-b8aa-6a155be721e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm\" (UID: \"11a79a51-82c7-4a50-b8aa-6a155be721e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"
Nov 02 09:54:45.491551 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.491355    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98dd9543-5592-461c-94e8-7fffe7079d7f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf\" (UID: \"98dd9543-5592-461c-94e8-7fffe7079d7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"
Nov 02 09:54:45.491551 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.491481    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqfjk\" (UniqueName: \"kubernetes.io/projected/98dd9543-5592-461c-94e8-7fffe7079d7f-kube-api-access-kqfjk\") pod \"payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf\" (UID: \"98dd9543-5592-461c-94e8-7fffe7079d7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"
Nov 02 09:54:45.495239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.493317    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a79a51-82c7-4a50-b8aa-6a155be721e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm\" (UID: \"11a79a51-82c7-4a50-b8aa-6a155be721e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"
Nov 02 09:54:45.495239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.494145    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98dd9543-5592-461c-94e8-7fffe7079d7f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf\" (UID: \"98dd9543-5592-461c-94e8-7fffe7079d7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"
Nov 02 09:54:45.535191 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.535152    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqfjk\" (UniqueName: \"kubernetes.io/projected/98dd9543-5592-461c-94e8-7fffe7079d7f-kube-api-access-kqfjk\") pod \"payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf\" (UID: \"98dd9543-5592-461c-94e8-7fffe7079d7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"
Nov 02 09:54:45.536942 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.536912    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq4dk\" (UniqueName: \"kubernetes.io/projected/11a79a51-82c7-4a50-b8aa-6a155be721e3-kube-api-access-tq4dk\") pod \"payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm\" (UID: \"11a79a51-82c7-4a50-b8aa-6a155be721e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"
Nov 02 09:54:45.629028 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.628974    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"
Nov 02 09:54:45.653457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.653420    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"
Nov 02 09:54:45.948267 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:54:45.948192    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98dd9543_5592_461c_94e8_7fffe7079d7f.slice/crio-395a2fec7982f2a9235d4039be7a1f6ee52b4bd2c96ce0bd02ad896a434000c4 WatchSource:0}: Error finding container 395a2fec7982f2a9235d4039be7a1f6ee52b4bd2c96ce0bd02ad896a434000c4: Status 404 returned error can't find the container with id 395a2fec7982f2a9235d4039be7a1f6ee52b4bd2c96ce0bd02ad896a434000c4
Nov 02 09:54:45.949975 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.949871    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"]
Nov 02 09:54:45.970971 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:54:45.970934    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a79a51_82c7_4a50_b8aa_6a155be721e3.slice/crio-8a97d14a25935233484d616865c9499d7d7d45701dc19d194ceb77c0b5122014 WatchSource:0}: Error finding container 8a97d14a25935233484d616865c9499d7d7d45701dc19d194ceb77c0b5122014: Status 404 returned error can't find the container with id 8a97d14a25935233484d616865c9499d7d7d45701dc19d194ceb77c0b5122014
Nov 02 09:54:45.973170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:45.973147    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"]
Nov 02 09:54:46.947775 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:46.947738    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf" event={"ID":"98dd9543-5592-461c-94e8-7fffe7079d7f","Type":"ContainerStarted","Data":"057dc6eac0ce8a0776dba08b7c4026d9aeb21c8037af7749952fae6aaf59aac8"}
Nov 02 09:54:46.947999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:46.947817    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"
Nov 02 09:54:46.947999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:46.947840    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf" event={"ID":"98dd9543-5592-461c-94e8-7fffe7079d7f","Type":"ContainerStarted","Data":"395a2fec7982f2a9235d4039be7a1f6ee52b4bd2c96ce0bd02ad896a434000c4"}
Nov 02 09:54:46.951737 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:46.951698    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm" event={"ID":"11a79a51-82c7-4a50-b8aa-6a155be721e3","Type":"ContainerStarted","Data":"39a141d5dc02c5deb5d456585a28bdd625042866fa971905e4d988b33863665a"}
Nov 02 09:54:46.951737 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:46.951769    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm" event={"ID":"11a79a51-82c7-4a50-b8aa-6a155be721e3","Type":"ContainerStarted","Data":"8a97d14a25935233484d616865c9499d7d7d45701dc19d194ceb77c0b5122014"}
Nov 02 09:54:46.951737 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:46.951794    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"
Nov 02 09:54:46.995254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:46.995184    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf" podStartSLOduration=1.995167962 podStartE2EDuration="1.995167962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:46.97247653 +0000 UTC m=+1163.910763303" watchObservedRunningTime="2026-11-02 09:54:46.995167962 +0000 UTC m=+1163.933454794"
Nov 02 09:54:46.995447 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:46.995365    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm" podStartSLOduration=1.995358967 podStartE2EDuration="1.995358967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:46.991359469 +0000 UTC m=+1163.929646247" watchObservedRunningTime="2026-11-02 09:54:46.995358967 +0000 UTC m=+1163.933645742"
Nov 02 09:54:52.323158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.323119    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs" podUID="4d268d7a-fd29-4177-93b1-f26c23409fec" containerName="main" containerID="cri-o://445e0209e77338f8a68a2e197536f0770792f4937bd3de4c492a51d298f1fc14" gracePeriod=45
Nov 02 09:54:52.417575 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.417537    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs" podUID="4d268d7a-fd29-4177-93b1-f26c23409fec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.157:8000/ready\": dial tcp 10.134.0.157:8000: connect: connection refused"
Nov 02 09:54:52.633071 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.633045    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:54:52.744988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.744957    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-dshm\") pod \"4d268d7a-fd29-4177-93b1-f26c23409fec\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") "
Nov 02 09:54:52.745137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.745009    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-tmp-dir\") pod \"4d268d7a-fd29-4177-93b1-f26c23409fec\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") "
Nov 02 09:54:52.745137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.745097    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d268d7a-fd29-4177-93b1-f26c23409fec-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d268d7a-fd29-4177-93b1-f26c23409fec-tls-certs\") pod \"4d268d7a-fd29-4177-93b1-f26c23409fec\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") "
Nov 02 09:54:52.745137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.745119    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d268d7a-fd29-4177-93b1-f26c23409fec-kube-api-access-tchtz\" (UniqueName: \"kubernetes.io/projected/4d268d7a-fd29-4177-93b1-f26c23409fec-kube-api-access-tchtz\") pod \"4d268d7a-fd29-4177-93b1-f26c23409fec\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") "
Nov 02 09:54:52.745322 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.745164    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-model-cache\") pod \"4d268d7a-fd29-4177-93b1-f26c23409fec\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") "
Nov 02 09:54:52.745322 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.745276    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-home\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-home\") pod \"4d268d7a-fd29-4177-93b1-f26c23409fec\" (UID: \"4d268d7a-fd29-4177-93b1-f26c23409fec\") "
Nov 02 09:54:52.745437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.745319    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-tmp-dir" pod "4d268d7a-fd29-4177-93b1-f26c23409fec" (UID: "4d268d7a-fd29-4177-93b1-f26c23409fec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:54:52.745530 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.745491    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-model-cache" pod "4d268d7a-fd29-4177-93b1-f26c23409fec" (UID: "4d268d7a-fd29-4177-93b1-f26c23409fec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:54:52.745581 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.745536    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-home" pod "4d268d7a-fd29-4177-93b1-f26c23409fec" (UID: "4d268d7a-fd29-4177-93b1-f26c23409fec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:54:52.746154 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.746130    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:52.746154 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.746151    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:52.746400 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.746167    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:52.747496 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.747473    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-dshm" pod "4d268d7a-fd29-4177-93b1-f26c23409fec" (UID: "4d268d7a-fd29-4177-93b1-f26c23409fec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:54:52.747607 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.747498    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d268d7a-fd29-4177-93b1-f26c23409fec-kube-api-access-tchtz" pod "4d268d7a-fd29-4177-93b1-f26c23409fec" (UID: "4d268d7a-fd29-4177-93b1-f26c23409fec"). InnerVolumeSpecName "kube-api-access-tchtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:52.747607 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.747558    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d268d7a-fd29-4177-93b1-f26c23409fec-tls-certs" pod "4d268d7a-fd29-4177-93b1-f26c23409fec" (UID: "4d268d7a-fd29-4177-93b1-f26c23409fec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:54:52.847833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.847800    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d268d7a-fd29-4177-93b1-f26c23409fec-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:52.847833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.847833    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tchtz\" (UniqueName: \"kubernetes.io/projected/4d268d7a-fd29-4177-93b1-f26c23409fec-kube-api-access-tchtz\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:52.848033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:52.847867    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d268d7a-fd29-4177-93b1-f26c23409fec-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:53.077305 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:53.077267    4449 generic.go:356] "Generic (PLEG): container finished" podID="4d268d7a-fd29-4177-93b1-f26c23409fec" containerID="445e0209e77338f8a68a2e197536f0770792f4937bd3de4c492a51d298f1fc14" exitCode=0
Nov 02 09:54:53.077516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:53.077349    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"
Nov 02 09:54:53.077516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:53.077355    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs" event={"ID":"4d268d7a-fd29-4177-93b1-f26c23409fec","Type":"ContainerDied","Data":"445e0209e77338f8a68a2e197536f0770792f4937bd3de4c492a51d298f1fc14"}
Nov 02 09:54:53.077516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:53.077408    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs" event={"ID":"4d268d7a-fd29-4177-93b1-f26c23409fec","Type":"ContainerDied","Data":"795b323967c21d0ad0710e04f8be98f4f77ba1fb0925dc4a960c6977c2da06b0"}
Nov 02 09:54:53.077516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:53.077430    4449 scope.go:122] "RemoveContainer" containerID="445e0209e77338f8a68a2e197536f0770792f4937bd3de4c492a51d298f1fc14"
Nov 02 09:54:53.091892 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:53.091871    4449 scope.go:122] "RemoveContainer" containerID="445e0209e77338f8a68a2e197536f0770792f4937bd3de4c492a51d298f1fc14"
Nov 02 09:54:53.092196 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:54:53.092176    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445e0209e77338f8a68a2e197536f0770792f4937bd3de4c492a51d298f1fc14\": container with ID starting with 445e0209e77338f8a68a2e197536f0770792f4937bd3de4c492a51d298f1fc14 not found: ID does not exist" containerID="445e0209e77338f8a68a2e197536f0770792f4937bd3de4c492a51d298f1fc14"
Nov 02 09:54:53.092351 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:53.092207    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445e0209e77338f8a68a2e197536f0770792f4937bd3de4c492a51d298f1fc14"} err="failed to get container status \"445e0209e77338f8a68a2e197536f0770792f4937bd3de4c492a51d298f1fc14\": rpc error: code = NotFound desc = could not find container \"445e0209e77338f8a68a2e197536f0770792f4937bd3de4c492a51d298f1fc14\": container with ID starting with 445e0209e77338f8a68a2e197536f0770792f4937bd3de4c492a51d298f1fc14 not found: ID does not exist"
Nov 02 09:54:53.106024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:53.105995    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"]
Nov 02 09:54:53.111376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:53.111352    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9cbd331c/e2e-lifecycle-model-9cbd331c-kserve-6656b467cc-5vbhs"]
Nov 02 09:54:53.208178 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:53.208146    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d268d7a-fd29-4177-93b1-f26c23409fec" path="/var/lib/kubelet/pods/4d268d7a-fd29-4177-93b1-f26c23409fec/volumes"
Nov 02 09:54:54.028434 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:54.028398    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"]
Nov 02 09:54:54.033939 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:54.033911    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c433d6fa-pggx5"]
Nov 02 09:54:55.207889 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:55.207854    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800" path="/var/lib/kubelet/pods/0d724aa4-01f5-4ae1-bdac-a0ec4dc9e800/volumes"
Nov 02 09:54:55.925291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:55.925265    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:54:56.031634 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:56.031599    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"]
Nov 02 09:54:56.036915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:56.036887    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-391e80b2-9czlt"]
Nov 02 09:54:57.120302 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.120272    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"]
Nov 02 09:54:57.121258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.121195    4449 cpu_manager.go:414] "containerMap: removing container" podUID="4d268d7a-fd29-4177-93b1-f26c23409fec" containerName="main"
Nov 02 09:54:57.121258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.121258    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d268d7a-fd29-4177-93b1-f26c23409fec" containerName="main"
Nov 02 09:54:57.121496 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.121481    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d268d7a-fd29-4177-93b1-f26c23409fec" containerName="main"
Nov 02 09:54:57.126914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.126892    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.138642 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.138615    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"]
Nov 02 09:54:57.209431 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.209397    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9366ff10-7d62-482f-a721-e626f53cf72d" path="/var/lib/kubelet/pods/9366ff10-7d62-482f-a721-e626f53cf72d/volumes"
Nov 02 09:54:57.227102 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.227059    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.227321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.227126    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.227321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.227230    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.227321 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.227273    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prn5v\" (UniqueName: \"kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-kube-api-access-prn5v\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.227498 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.227409    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.227568 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.227546    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.227646 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.227631    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.227707 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.227659    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.227774 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.227712    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.329369 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.329326    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.329558 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.329436    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.329558 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.329481    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.329558 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.329523    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prn5v\" (UniqueName: \"kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-kube-api-access-prn5v\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.329781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.329642    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.329842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.329789    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.329842 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.329796    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.329956 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.329900    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.329956 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.329935    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.330060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.330008    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.330113 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.330087    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.330476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.330413    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.330650 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.330595    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.331011 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.330988    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.332453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.332430    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.333447 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.333424    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.343269 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.343205    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.344024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.344001    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prn5v\" (UniqueName: \"kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-kube-api-access-prn5v\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.445899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.445805    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:57.764908 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:54:57.764869    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220a9b39_e990_44ed_bdf7_4c039fa4a9ec.slice/crio-db1c0083c394d9c77d17aee1e0cf779aab8820a7f40d10bf9de703b6c78fdcc6 WatchSource:0}: Error finding container db1c0083c394d9c77d17aee1e0cf779aab8820a7f40d10bf9de703b6c78fdcc6: Status 404 returned error can't find the container with id db1c0083c394d9c77d17aee1e0cf779aab8820a7f40d10bf9de703b6c78fdcc6
Nov 02 09:54:57.766380 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.766353    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"]
Nov 02 09:54:57.767572 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.767541    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:57.767645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.767605    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:57.767645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.767632    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:54:57.964914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.964884    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"
Nov 02 09:54:57.965089 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:57.964965    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"
Nov 02 09:54:58.152071 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:58.152033    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f" event={"ID":"220a9b39-e990-44ed-bdf7-4c039fa4a9ec","Type":"ContainerStarted","Data":"dc48825f6451ccdc587ee88c6d5c207a70a0b4deefadf8dd3566e45e84f40832"}
Nov 02 09:54:58.152633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:58.152100    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f" event={"ID":"220a9b39-e990-44ed-bdf7-4c039fa4a9ec","Type":"ContainerStarted","Data":"db1c0083c394d9c77d17aee1e0cf779aab8820a7f40d10bf9de703b6c78fdcc6"}
Nov 02 09:54:58.177888 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:58.177837    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f" podStartSLOduration=1.177823525 podStartE2EDuration="1.177823525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:58.173400965 +0000 UTC m=+1175.111687771" watchObservedRunningTime="2026-11-02 09:54:58.177823525 +0000 UTC m=+1175.116110300"
Nov 02 09:54:58.446463 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:58.446385    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:54:58.447826 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:58.447797    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 02 09:54:58.447988 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:58.447846    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f" podUID="220a9b39-e990-44ed-bdf7-4c039fa4a9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 02 09:54:59.446967 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:59.446930    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 02 09:54:59.447484 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:54:59.447006    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f" podUID="220a9b39-e990-44ed-bdf7-4c039fa4a9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 02 09:55:00.446615 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:00.446582    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 02 09:55:00.446798 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:00.446637    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f" podUID="220a9b39-e990-44ed-bdf7-4c039fa4a9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 02 09:55:01.446923 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:01.446885    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 02 09:55:01.447433 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:01.446982    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f" podUID="220a9b39-e990-44ed-bdf7-4c039fa4a9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 02 09:55:02.446496 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:02.446462    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 02 09:55:02.446675 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:02.446527    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f" podUID="220a9b39-e990-44ed-bdf7-4c039fa4a9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 02 09:55:03.446966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:03.446937    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 02 09:55:03.447420 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:03.446997    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f" podUID="220a9b39-e990-44ed-bdf7-4c039fa4a9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 02 09:55:04.586742 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:04.586708    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:55:04.587633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:04.587603    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:55:04.588608 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:04.588582    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:55:06.549413 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:55:06.549374    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:55:07.028290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:07.028197    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"]
Nov 02 09:55:07.033514 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:07.033486    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd856d0-xpfkn"]
Nov 02 09:55:07.211670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:07.211621    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257" path="/var/lib/kubelet/pods/e9f3d8b7-c0f1-4f6d-bd0a-a6750c24a257/volumes"
Nov 02 09:55:12.641525 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:12.641483    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"]
Nov 02 09:55:14.842003 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:14.841965    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"]
Nov 02 09:55:14.856395 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:14.856358    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"]
Nov 02 09:55:17.642093 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:17.642028    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k" podUID="b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99" containerName="payload-processing" containerID="cri-o://a526d6b9e1dcf59ead21fe8a8da6a316bac547c34fb7016018282324d706539b" gracePeriod=25
Nov 02 09:55:17.954977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:17.954946    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"
Nov 02 09:55:18.093034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.092994    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-kube-api-access-qknx5\" (UniqueName: \"kubernetes.io/projected/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-kube-api-access-qknx5\") pod \"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99\" (UID: \"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99\") "
Nov 02 09:55:18.093284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.093066    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-plugins-config-volume\") pod \"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99\" (UID: \"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99\") "
Nov 02 09:55:18.093903 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.093868    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-plugins-config-volume" pod "b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99" (UID: "b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:18.096069 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.096038    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-kube-api-access-qknx5" pod "b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99" (UID: "b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99"). InnerVolumeSpecName "kube-api-access-qknx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:18.194787 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.194697    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qknx5\" (UniqueName: \"kubernetes.io/projected/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-kube-api-access-qknx5\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:18.194787 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.194731    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:18.519201 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.519081    4449 generic.go:356] "Generic (PLEG): container finished" podID="b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99" containerID="a526d6b9e1dcf59ead21fe8a8da6a316bac547c34fb7016018282324d706539b" exitCode=0
Nov 02 09:55:18.519504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.519273    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k" event={"ID":"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99","Type":"ContainerDied","Data":"a526d6b9e1dcf59ead21fe8a8da6a316bac547c34fb7016018282324d706539b"}
Nov 02 09:55:18.519504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.519342    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k" event={"ID":"b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99","Type":"ContainerDied","Data":"143c8272d86d8f6ae91db08519817282489110adcb1321873d78948305c84355"}
Nov 02 09:55:18.519504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.519369    4449 scope.go:122] "RemoveContainer" containerID="a526d6b9e1dcf59ead21fe8a8da6a316bac547c34fb7016018282324d706539b"
Nov 02 09:55:18.519689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.519530    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"
Nov 02 09:55:18.535901 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.535877    4449 scope.go:122] "RemoveContainer" containerID="a526d6b9e1dcf59ead21fe8a8da6a316bac547c34fb7016018282324d706539b"
Nov 02 09:55:18.536296 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:55:18.536272    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a526d6b9e1dcf59ead21fe8a8da6a316bac547c34fb7016018282324d706539b\": container with ID starting with a526d6b9e1dcf59ead21fe8a8da6a316bac547c34fb7016018282324d706539b not found: ID does not exist" containerID="a526d6b9e1dcf59ead21fe8a8da6a316bac547c34fb7016018282324d706539b"
Nov 02 09:55:18.536361 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.536305    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a526d6b9e1dcf59ead21fe8a8da6a316bac547c34fb7016018282324d706539b"} err="failed to get container status \"a526d6b9e1dcf59ead21fe8a8da6a316bac547c34fb7016018282324d706539b\": rpc error: code = NotFound desc = could not find container \"a526d6b9e1dcf59ead21fe8a8da6a316bac547c34fb7016018282324d706539b\": container with ID starting with a526d6b9e1dcf59ead21fe8a8da6a316bac547c34fb7016018282324d706539b not found: ID does not exist"
Nov 02 09:55:18.550819 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.550782    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"]
Nov 02 09:55:18.556395 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:18.556366    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ced1b7-865949c6d5-w8d7k"]
Nov 02 09:55:19.211948 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:19.211910    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99" path="/var/lib/kubelet/pods/b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99/volumes"
Nov 02 09:55:19.844001 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:19.843955    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn" podUID="12cade1c-2ef3-4274-88ef-fe9686d7064d" containerName="payload-processing" containerID="cri-o://53dd3108f1efb67ec157fbe2c9a90c8654d337541006f4c4f90912d8dd983a76" gracePeriod=25
Nov 02 09:55:19.860298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:19.857974    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7" podUID="5ed5058e-bb88-45a0-a231-198f4214bf7c" containerName="payload-pre-processing" containerID="cri-o://e9424cab4f6d2f3bdf195cd5b9e3275e7913bf2144469d5fad9c2e9efef745ea" gracePeriod=25
Nov 02 09:55:20.384528 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.384498    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"
Nov 02 09:55:20.388719 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.388693    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"
Nov 02 09:55:20.538288 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.538255    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12cade1c-2ef3-4274-88ef-fe9686d7064d-kube-api-access-h9mzs\" (UniqueName: \"kubernetes.io/projected/12cade1c-2ef3-4274-88ef-fe9686d7064d-kube-api-access-h9mzs\") pod \"12cade1c-2ef3-4274-88ef-fe9686d7064d\" (UID: \"12cade1c-2ef3-4274-88ef-fe9686d7064d\") "
Nov 02 09:55:20.538500 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.538471    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12cade1c-2ef3-4274-88ef-fe9686d7064d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cade1c-2ef3-4274-88ef-fe9686d7064d-plugins-config-volume\") pod \"12cade1c-2ef3-4274-88ef-fe9686d7064d\" (UID: \"12cade1c-2ef3-4274-88ef-fe9686d7064d\") "
Nov 02 09:55:20.538574 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.538546    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed5058e-bb88-45a0-a231-198f4214bf7c-kube-api-access-bvnp8\" (UniqueName: \"kubernetes.io/projected/5ed5058e-bb88-45a0-a231-198f4214bf7c-kube-api-access-bvnp8\") pod \"5ed5058e-bb88-45a0-a231-198f4214bf7c\" (UID: \"5ed5058e-bb88-45a0-a231-198f4214bf7c\") "
Nov 02 09:55:20.538652 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.538639    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed5058e-bb88-45a0-a231-198f4214bf7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed5058e-bb88-45a0-a231-198f4214bf7c-plugins-config-volume\") pod \"5ed5058e-bb88-45a0-a231-198f4214bf7c\" (UID: \"5ed5058e-bb88-45a0-a231-198f4214bf7c\") "
Nov 02 09:55:20.538901 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.538874    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12cade1c-2ef3-4274-88ef-fe9686d7064d-plugins-config-volume" pod "12cade1c-2ef3-4274-88ef-fe9686d7064d" (UID: "12cade1c-2ef3-4274-88ef-fe9686d7064d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:20.539388 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.539289    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed5058e-bb88-45a0-a231-198f4214bf7c-plugins-config-volume" pod "5ed5058e-bb88-45a0-a231-198f4214bf7c" (UID: "5ed5058e-bb88-45a0-a231-198f4214bf7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:20.539565 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.539465    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cade1c-2ef3-4274-88ef-fe9686d7064d-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:20.539565 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.539488    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed5058e-bb88-45a0-a231-198f4214bf7c-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:20.540790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.540757    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cade1c-2ef3-4274-88ef-fe9686d7064d-kube-api-access-h9mzs" pod "12cade1c-2ef3-4274-88ef-fe9686d7064d" (UID: "12cade1c-2ef3-4274-88ef-fe9686d7064d"). InnerVolumeSpecName "kube-api-access-h9mzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:20.541075 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.541051    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed5058e-bb88-45a0-a231-198f4214bf7c-kube-api-access-bvnp8" pod "5ed5058e-bb88-45a0-a231-198f4214bf7c" (UID: "5ed5058e-bb88-45a0-a231-198f4214bf7c"). InnerVolumeSpecName "kube-api-access-bvnp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:20.542426 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.542403    4449 generic.go:356] "Generic (PLEG): container finished" podID="12cade1c-2ef3-4274-88ef-fe9686d7064d" containerID="53dd3108f1efb67ec157fbe2c9a90c8654d337541006f4c4f90912d8dd983a76" exitCode=0
Nov 02 09:55:20.542569 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.542461    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"
Nov 02 09:55:20.542569 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.542487    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn" event={"ID":"12cade1c-2ef3-4274-88ef-fe9686d7064d","Type":"ContainerDied","Data":"53dd3108f1efb67ec157fbe2c9a90c8654d337541006f4c4f90912d8dd983a76"}
Nov 02 09:55:20.542569 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.542550    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn" event={"ID":"12cade1c-2ef3-4274-88ef-fe9686d7064d","Type":"ContainerDied","Data":"7008e3e5076c7aaa210f68aa6be47f94e13fa9bad6833e7245c2b3d14f9b085f"}
Nov 02 09:55:20.542737 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.542580    4449 scope.go:122] "RemoveContainer" containerID="53dd3108f1efb67ec157fbe2c9a90c8654d337541006f4c4f90912d8dd983a76"
Nov 02 09:55:20.544407 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.544384    4449 generic.go:356] "Generic (PLEG): container finished" podID="5ed5058e-bb88-45a0-a231-198f4214bf7c" containerID="e9424cab4f6d2f3bdf195cd5b9e3275e7913bf2144469d5fad9c2e9efef745ea" exitCode=0
Nov 02 09:55:20.544523 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.544450    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"
Nov 02 09:55:20.544523 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.544467    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7" event={"ID":"5ed5058e-bb88-45a0-a231-198f4214bf7c","Type":"ContainerDied","Data":"e9424cab4f6d2f3bdf195cd5b9e3275e7913bf2144469d5fad9c2e9efef745ea"}
Nov 02 09:55:20.544603 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.544526    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7" event={"ID":"5ed5058e-bb88-45a0-a231-198f4214bf7c","Type":"ContainerDied","Data":"adbca93c3c9b9bf390d6862955d8a0c677785a5a743d5997dfd4a575bffc7903"}
Nov 02 09:55:20.564622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.564593    4449 scope.go:122] "RemoveContainer" containerID="53dd3108f1efb67ec157fbe2c9a90c8654d337541006f4c4f90912d8dd983a76"
Nov 02 09:55:20.564990 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:55:20.564968    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53dd3108f1efb67ec157fbe2c9a90c8654d337541006f4c4f90912d8dd983a76\": container with ID starting with 53dd3108f1efb67ec157fbe2c9a90c8654d337541006f4c4f90912d8dd983a76 not found: ID does not exist" containerID="53dd3108f1efb67ec157fbe2c9a90c8654d337541006f4c4f90912d8dd983a76"
Nov 02 09:55:20.565064 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.564999    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53dd3108f1efb67ec157fbe2c9a90c8654d337541006f4c4f90912d8dd983a76"} err="failed to get container status \"53dd3108f1efb67ec157fbe2c9a90c8654d337541006f4c4f90912d8dd983a76\": rpc error: code = NotFound desc = could not find container \"53dd3108f1efb67ec157fbe2c9a90c8654d337541006f4c4f90912d8dd983a76\": container with ID starting with 53dd3108f1efb67ec157fbe2c9a90c8654d337541006f4c4f90912d8dd983a76 not found: ID does not exist"
Nov 02 09:55:20.565064 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.565019    4449 scope.go:122] "RemoveContainer" containerID="e9424cab4f6d2f3bdf195cd5b9e3275e7913bf2144469d5fad9c2e9efef745ea"
Nov 02 09:55:20.578823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.578802    4449 scope.go:122] "RemoveContainer" containerID="e9424cab4f6d2f3bdf195cd5b9e3275e7913bf2144469d5fad9c2e9efef745ea"
Nov 02 09:55:20.579162 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:55:20.579138    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9424cab4f6d2f3bdf195cd5b9e3275e7913bf2144469d5fad9c2e9efef745ea\": container with ID starting with e9424cab4f6d2f3bdf195cd5b9e3275e7913bf2144469d5fad9c2e9efef745ea not found: ID does not exist" containerID="e9424cab4f6d2f3bdf195cd5b9e3275e7913bf2144469d5fad9c2e9efef745ea"
Nov 02 09:55:20.579258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.579160    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"]
Nov 02 09:55:20.579258 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.579173    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9424cab4f6d2f3bdf195cd5b9e3275e7913bf2144469d5fad9c2e9efef745ea"} err="failed to get container status \"e9424cab4f6d2f3bdf195cd5b9e3275e7913bf2144469d5fad9c2e9efef745ea\": rpc error: code = NotFound desc = could not find container \"e9424cab4f6d2f3bdf195cd5b9e3275e7913bf2144469d5fad9c2e9efef745ea\": container with ID starting with e9424cab4f6d2f3bdf195cd5b9e3275e7913bf2144469d5fad9c2e9efef745ea not found: ID does not exist"
Nov 02 09:55:20.584672 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.584638    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9cbd331c-789b6c9bcf-v4jgn"]
Nov 02 09:55:20.591554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.591520    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"]
Nov 02 09:55:20.596940 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.596911    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9cbd331c-b5c5f659-tm6n7"]
Nov 02 09:55:20.641067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.641035    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9mzs\" (UniqueName: \"kubernetes.io/projected/12cade1c-2ef3-4274-88ef-fe9686d7064d-kube-api-access-h9mzs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:20.641067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:20.641065    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvnp8\" (UniqueName: \"kubernetes.io/projected/5ed5058e-bb88-45a0-a231-198f4214bf7c-kube-api-access-bvnp8\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:21.210319 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:21.210184    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12cade1c-2ef3-4274-88ef-fe9686d7064d" path="/var/lib/kubelet/pods/12cade1c-2ef3-4274-88ef-fe9686d7064d/volumes"
Nov 02 09:55:21.210749 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:21.210734    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed5058e-bb88-45a0-a231-198f4214bf7c" path="/var/lib/kubelet/pods/5ed5058e-bb88-45a0-a231-198f4214bf7c/volumes"
Nov 02 09:55:22.546553 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.546524    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"]
Nov 02 09:55:22.547262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.547247    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99" containerName="payload-processing"
Nov 02 09:55:22.547262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.547263    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99" containerName="payload-processing"
Nov 02 09:55:22.547356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.547282    4449 cpu_manager.go:414] "containerMap: removing container" podUID="12cade1c-2ef3-4274-88ef-fe9686d7064d" containerName="payload-processing"
Nov 02 09:55:22.547356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.547287    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12cade1c-2ef3-4274-88ef-fe9686d7064d" containerName="payload-processing"
Nov 02 09:55:22.547356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.547308    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5ed5058e-bb88-45a0-a231-198f4214bf7c" containerName="payload-pre-processing"
Nov 02 09:55:22.547356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.547313    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed5058e-bb88-45a0-a231-198f4214bf7c" containerName="payload-pre-processing"
Nov 02 09:55:22.547481 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.547424    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a2e7a6-b8f8-45d3-8d54-614f7ba17b99" containerName="payload-processing"
Nov 02 09:55:22.547481 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.547434    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="12cade1c-2ef3-4274-88ef-fe9686d7064d" containerName="payload-processing"
Nov 02 09:55:22.547481 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.547443    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed5058e-bb88-45a0-a231-198f4214bf7c" containerName="payload-pre-processing"
Nov 02 09:55:22.552924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.552899    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.565682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.565645    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-9a69d1-openshift-default-dockercfg-n9g8z\""
Nov 02 09:55:22.574152 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.574122    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"]
Nov 02 09:55:22.682306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.682268    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istiod-ca-cert\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.682306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.682314    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-data\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.682530 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.682468    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96v6s\" (UniqueName: \"kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-kube-api-access-96v6s\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.682530 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.682506    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-envoy\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.682618 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.682531    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-certs\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.682670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.682650    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-podinfo\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.682717 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.682704    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-credential-socket\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.682846 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.682834    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-token\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.682885 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.682860    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-socket\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.810291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.804583    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istiod-ca-cert\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.810291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.804638    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-data\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.810291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.804753    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96v6s\" (UniqueName: \"kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-kube-api-access-96v6s\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.810291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.804794    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-envoy\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.810291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.804824    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-certs\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.810291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.804878    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-podinfo\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.810291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.804919    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-credential-socket\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.810291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.805092    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-token\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.810291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.805125    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-socket\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.810291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.805663    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-socket\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.810291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.806344    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istiod-ca-cert\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.810291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.806673    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-data\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.815240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.812142    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-credential-socket\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.815240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.812445    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-certs\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.867236 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.844596    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96v6s\" (UniqueName: \"kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-kube-api-access-96v6s\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.867236 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.851027    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-podinfo\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.878236 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.875829    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-envoy\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:22.878236 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:22.877418    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-token\") pod \"e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") " pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:23.170780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:23.170698    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:23.478514 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:55:23.478476    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa471864_8cf5_4032_8a6d_c6c61eece6fc.slice/crio-404077af4a2888bf3f29b32a0f8c0cf5564524e3abdd275356b2aabae1d242d7 WatchSource:0}: Error finding container 404077af4a2888bf3f29b32a0f8c0cf5564524e3abdd275356b2aabae1d242d7: Status 404 returned error can't find the container with id 404077af4a2888bf3f29b32a0f8c0cf5564524e3abdd275356b2aabae1d242d7
Nov 02 09:55:23.481634 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:23.480834    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"]
Nov 02 09:55:23.492972 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:23.492924    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:55:23.493145 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:23.493021    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:55:23.493145 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:23.493063    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:55:23.585307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:23.585205    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" event={"ID":"fa471864-8cf5-4032-8a6d-c6c61eece6fc","Type":"ContainerStarted","Data":"404077af4a2888bf3f29b32a0f8c0cf5564524e3abdd275356b2aabae1d242d7"}
Nov 02 09:55:23.748078 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:23.747977    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"]
Nov 02 09:55:23.748598 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:23.748568    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy" containerID="cri-o://8cc48100801023880a7d78c42a0f546872a153e2b62a66eeef3adb335aa1e692" gracePeriod=30
Nov 02 09:55:24.067297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:24.052620    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"]
Nov 02 09:55:24.093044 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:24.092990    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe06209-72gd6"]
Nov 02 09:55:24.262109 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:24.262070    4449 scope.go:122] "RemoveContainer" containerID="575cac6caa5982a6c4fb4239c68c9b065a078f41396381c311db9391663a088a"
Nov 02 09:55:24.275371 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:24.275344    4449 scope.go:122] "RemoveContainer" containerID="e07064db4c1e99227ac245261b09566f26480d8f84f5ac7d8447782e4d1c5b38"
Nov 02 09:55:24.287108 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:24.287090    4449 scope.go:122] "RemoveContainer" containerID="e6817f9b7dbc3f7445f9367ca84f7ea4f716b08491dc010d862eb7540f5b7bbf"
Nov 02 09:55:24.299103 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:24.299079    4449 scope.go:122] "RemoveContainer" containerID="ece06224952f564bfc4446fd64441e0c835954693e865fdaab75feb1677adfbb"
Nov 02 09:55:24.310949 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:24.310927    4449 scope.go:122] "RemoveContainer" containerID="65f49c7709a70ccad45dec5858d862ae336ee26feaf9c8e5a242968fa198c468"
Nov 02 09:55:24.607345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:24.606944    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" event={"ID":"fa471864-8cf5-4032-8a6d-c6c61eece6fc","Type":"ContainerStarted","Data":"9c05a6a210cbcd646920b0281a8e82733cbace7b3af3610e1a2f02ab3bff1539"}
Nov 02 09:55:24.648576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:24.648497    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" podStartSLOduration=2.648470392 podStartE2EDuration="2.648470392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:24.643675006 +0000 UTC m=+1201.581961784" watchObservedRunningTime="2026-11-02 09:55:24.648470392 +0000 UTC m=+1201.586757173"
Nov 02 09:55:25.139107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.139066    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"]
Nov 02 09:55:25.146155 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.146117    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"
Nov 02 09:55:25.151735 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.151702    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:55:25.151735 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.151702    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:55:25.151945 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.151824    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:55:25.151945 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.151888    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:55:25.154563 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.154539    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"]
Nov 02 09:55:25.170943 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.170910    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:25.176835 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.176805    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:25.214835 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.214797    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79021ba5-953f-461e-b75e-957242a29b3a" path="/var/lib/kubelet/pods/79021ba5-953f-461e-b75e-957242a29b3a/volumes"
Nov 02 09:55:25.273990 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.273946    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d76ecdd-d7rmq\" (UID: \"230a22f3-5692-4bf8-8897-850431c806ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"
Nov 02 09:55:25.274484 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.274462    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ld7q\" (UniqueName: \"kubernetes.io/projected/230a22f3-5692-4bf8-8897-850431c806ec-kube-api-access-4ld7q\") pod \"maas-api-revoke-sub-9d76ecdd-d7rmq\" (UID: \"230a22f3-5692-4bf8-8897-850431c806ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"
Nov 02 09:55:25.274603 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.274516    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d76ecdd-d7rmq\" (UID: \"230a22f3-5692-4bf8-8897-850431c806ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"
Nov 02 09:55:25.384201 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.384158    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ld7q\" (UniqueName: \"kubernetes.io/projected/230a22f3-5692-4bf8-8897-850431c806ec-kube-api-access-4ld7q\") pod \"maas-api-revoke-sub-9d76ecdd-d7rmq\" (UID: \"230a22f3-5692-4bf8-8897-850431c806ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"
Nov 02 09:55:25.384935 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.384906    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d76ecdd-d7rmq\" (UID: \"230a22f3-5692-4bf8-8897-850431c806ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"
Nov 02 09:55:25.386360 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.386139    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d76ecdd-d7rmq\" (UID: \"230a22f3-5692-4bf8-8897-850431c806ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"
Nov 02 09:55:25.386520 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.386326    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d76ecdd-d7rmq\" (UID: \"230a22f3-5692-4bf8-8897-850431c806ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"
Nov 02 09:55:25.389507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.389444    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d76ecdd-d7rmq\" (UID: \"230a22f3-5692-4bf8-8897-850431c806ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"
Nov 02 09:55:25.399931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.399895    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ld7q\" (UniqueName: \"kubernetes.io/projected/230a22f3-5692-4bf8-8897-850431c806ec-kube-api-access-4ld7q\") pod \"maas-api-revoke-sub-9d76ecdd-d7rmq\" (UID: \"230a22f3-5692-4bf8-8897-850431c806ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"
Nov 02 09:55:25.466414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.466368    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"
Nov 02 09:55:25.621516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.621469    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:25.622699 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.622669    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:55:25.765094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:25.765061    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"]
Nov 02 09:55:26.631804 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:26.631768    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq" event={"ID":"230a22f3-5692-4bf8-8897-850431c806ec","Type":"ContainerStarted","Data":"e4f76e78cc47bdaae3856dcb93fe5b48fba573e60ba6d0f1d064206afe50fcf3"}
Nov 02 09:55:26.632231 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:26.631821    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq" event={"ID":"230a22f3-5692-4bf8-8897-850431c806ec","Type":"ContainerStarted","Data":"6a0356da29de34c6f45cbfb830474b4ef452bbaa17fe8d2ea2dfb4f74c6418e7"}
Nov 02 09:55:26.654033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:26.653985    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq" podStartSLOduration=1.653972489 podStartE2EDuration="1.653972489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:26.649199511 +0000 UTC m=+1203.587486301" watchObservedRunningTime="2026-11-02 09:55:26.653972489 +0000 UTC m=+1203.592259264"
Nov 02 09:55:27.643458 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:27.643425    4449 generic.go:356] "Generic (PLEG): container finished" podID="230a22f3-5692-4bf8-8897-850431c806ec" containerID="e4f76e78cc47bdaae3856dcb93fe5b48fba573e60ba6d0f1d064206afe50fcf3" exitCode=0
Nov 02 09:55:27.643958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:27.643525    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq" event={"ID":"230a22f3-5692-4bf8-8897-850431c806ec","Type":"ContainerDied","Data":"e4f76e78cc47bdaae3856dcb93fe5b48fba573e60ba6d0f1d064206afe50fcf3"}
Nov 02 09:55:28.889074 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:28.889045    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"
Nov 02 09:55:29.058760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.058730    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:55:29.062027 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.062000    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230a22f3-5692-4bf8-8897-850431c806ec-kube-api-access-4ld7q\" (UniqueName: \"kubernetes.io/projected/230a22f3-5692-4bf8-8897-850431c806ec-kube-api-access-4ld7q\") pod \"230a22f3-5692-4bf8-8897-850431c806ec\" (UID: \"230a22f3-5692-4bf8-8897-850431c806ec\") "
Nov 02 09:55:29.062230 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.062114    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-openshift-ca\") pod \"230a22f3-5692-4bf8-8897-850431c806ec\" (UID: \"230a22f3-5692-4bf8-8897-850431c806ec\") "
Nov 02 09:55:29.062366 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.062350    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-xks-ca\") pod \"230a22f3-5692-4bf8-8897-850431c806ec\" (UID: \"230a22f3-5692-4bf8-8897-850431c806ec\") "
Nov 02 09:55:29.062570 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.062541    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-openshift-ca" pod "230a22f3-5692-4bf8-8897-850431c806ec" (UID: "230a22f3-5692-4bf8-8897-850431c806ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:29.063057 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.063035    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.065084 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.065057    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-xks-ca" pod "230a22f3-5692-4bf8-8897-850431c806ec" (UID: "230a22f3-5692-4bf8-8897-850431c806ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:55:29.065194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.065085    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230a22f3-5692-4bf8-8897-850431c806ec-kube-api-access-4ld7q" pod "230a22f3-5692-4bf8-8897-850431c806ec" (UID: "230a22f3-5692-4bf8-8897-850431c806ec"). InnerVolumeSpecName "kube-api-access-4ld7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:29.164108 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.164069    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-podinfo\") pod \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") "
Nov 02 09:55:29.164385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.164129    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-credential-socket\") pod \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") "
Nov 02 09:55:29.164385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.164182    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-kube-api-access-nqzhk\" (UniqueName: \"kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-kube-api-access-nqzhk\") pod \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") "
Nov 02 09:55:29.164385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.164344    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-token\" (UniqueName: \"kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-token\") pod \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") "
Nov 02 09:55:29.164592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.164433    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-envoy\") pod \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") "
Nov 02 09:55:29.164592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.164519    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-data\") pod \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") "
Nov 02 09:55:29.164592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.164547    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-credential-socket" pod "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" (UID: "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:29.164592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.164570    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-socket\") pod \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") "
Nov 02 09:55:29.164792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.164623    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istiod-ca-cert\") pod \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") "
Nov 02 09:55:29.164792 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.164665    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-certs\") pod \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\" (UID: \"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc\") "
Nov 02 09:55:29.164906 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.164783    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-data" pod "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" (UID: "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:29.164906 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.164853    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-socket" pod "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" (UID: "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:29.165399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.165373    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-certs" pod "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" (UID: "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:29.165507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.165402    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istiod-ca-cert" pod "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" (UID: "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:29.165507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.165428    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ld7q\" (UniqueName: \"kubernetes.io/projected/230a22f3-5692-4bf8-8897-850431c806ec-kube-api-access-4ld7q\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.165507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.165450    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.165507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.165468    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.165507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.165483    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230a22f3-5692-4bf8-8897-850431c806ec-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.165507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.165496    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.166752 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.166727    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-envoy" pod "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" (UID: "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:29.166852 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.166771    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-kube-api-access-nqzhk" pod "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" (UID: "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc"). InnerVolumeSpecName "kube-api-access-nqzhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:29.167134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.167109    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-podinfo" pod "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" (UID: "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:55:29.167134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.167125    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-token" pod "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" (UID: "02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:29.276130 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.276094    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.276130 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.276128    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.276130 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.276138    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.276666 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.276148    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqzhk\" (UniqueName: \"kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-kube-api-access-nqzhk\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.276666 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.276157    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.276666 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.276166    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.682476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.682444    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"
Nov 02 09:55:29.682476 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.682477    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq" event={"ID":"230a22f3-5692-4bf8-8897-850431c806ec","Type":"ContainerDied","Data":"6a0356da29de34c6f45cbfb830474b4ef452bbaa17fe8d2ea2dfb4f74c6418e7"}
Nov 02 09:55:29.682777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.682559    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a0356da29de34c6f45cbfb830474b4ef452bbaa17fe8d2ea2dfb4f74c6418e7"
Nov 02 09:55:29.685172 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.685142    4449 generic.go:356] "Generic (PLEG): container finished" podID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerID="8cc48100801023880a7d78c42a0f546872a153e2b62a66eeef3adb335aa1e692" exitCode=0
Nov 02 09:55:29.685368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.685228    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"
Nov 02 09:55:29.685368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.685272    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" event={"ID":"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc","Type":"ContainerDied","Data":"8cc48100801023880a7d78c42a0f546872a153e2b62a66eeef3adb335aa1e692"}
Nov 02 09:55:29.685368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.685339    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7" event={"ID":"02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc","Type":"ContainerDied","Data":"be71c842a63ef67cde1b312242430ac9908601f607ccc3887917848a355b3f3b"}
Nov 02 09:55:29.685368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.685366    4449 scope.go:122] "RemoveContainer" containerID="8cc48100801023880a7d78c42a0f546872a153e2b62a66eeef3adb335aa1e692"
Nov 02 09:55:29.700811 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.700786    4449 scope.go:122] "RemoveContainer" containerID="85827e0be84c335ba8f2d9ee3460f4aa50669a03f14e148a739074408c35331b"
Nov 02 09:55:29.733262 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.725986    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"]
Nov 02 09:55:29.808200 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.807685    4449 scope.go:122] "RemoveContainer" containerID="8cc48100801023880a7d78c42a0f546872a153e2b62a66eeef3adb335aa1e692"
Nov 02 09:55:29.808200 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.808125    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-8pgt7"]
Nov 02 09:55:29.808639 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:55:29.808595    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc48100801023880a7d78c42a0f546872a153e2b62a66eeef3adb335aa1e692\": container with ID starting with 8cc48100801023880a7d78c42a0f546872a153e2b62a66eeef3adb335aa1e692 not found: ID does not exist" containerID="8cc48100801023880a7d78c42a0f546872a153e2b62a66eeef3adb335aa1e692"
Nov 02 09:55:29.808798 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.808768    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc48100801023880a7d78c42a0f546872a153e2b62a66eeef3adb335aa1e692"} err="failed to get container status \"8cc48100801023880a7d78c42a0f546872a153e2b62a66eeef3adb335aa1e692\": rpc error: code = NotFound desc = could not find container \"8cc48100801023880a7d78c42a0f546872a153e2b62a66eeef3adb335aa1e692\": container with ID starting with 8cc48100801023880a7d78c42a0f546872a153e2b62a66eeef3adb335aa1e692 not found: ID does not exist"
Nov 02 09:55:29.808932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.808918    4449 scope.go:122] "RemoveContainer" containerID="85827e0be84c335ba8f2d9ee3460f4aa50669a03f14e148a739074408c35331b"
Nov 02 09:55:29.809626 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:55:29.809604    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85827e0be84c335ba8f2d9ee3460f4aa50669a03f14e148a739074408c35331b\": container with ID starting with 85827e0be84c335ba8f2d9ee3460f4aa50669a03f14e148a739074408c35331b not found: ID does not exist" containerID="85827e0be84c335ba8f2d9ee3460f4aa50669a03f14e148a739074408c35331b"
Nov 02 09:55:29.829507 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:29.829451    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85827e0be84c335ba8f2d9ee3460f4aa50669a03f14e148a739074408c35331b"} err="failed to get container status \"85827e0be84c335ba8f2d9ee3460f4aa50669a03f14e148a739074408c35331b\": rpc error: code = NotFound desc = could not find container \"85827e0be84c335ba8f2d9ee3460f4aa50669a03f14e148a739074408c35331b\": container with ID starting with 85827e0be84c335ba8f2d9ee3460f4aa50669a03f14e148a739074408c35331b not found: ID does not exist"
Nov 02 09:55:31.209691 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:31.209651    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" path="/var/lib/kubelet/pods/02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc/volumes"
Nov 02 09:55:33.638440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:33.638399    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-24rkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:55:33.638949 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:33.638484    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:34.034323 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:34.032208    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"]
Nov 02 09:55:34.037945 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:34.037914    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"]
Nov 02 09:55:34.056374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:34.055949    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67af8aeb-qtq7g"]
Nov 02 09:55:34.058065 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:34.058026    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4a2740-d7p2s"]
Nov 02 09:55:35.208208 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:35.208106    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41" path="/var/lib/kubelet/pods/0dbc7ca7-5a25-4c72-af8d-a6f9747b2b41/volumes"
Nov 02 09:55:35.208855 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:35.208832    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a415d7-2ebb-48ed-a643-550892ef3774" path="/var/lib/kubelet/pods/10a415d7-2ebb-48ed-a643-550892ef3774/volumes"
Nov 02 09:55:36.552347 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:55:36.552309    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:55:37.900280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:37.900245    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"]
Nov 02 09:55:37.901233 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:37.901187    4449 cpu_manager.go:414] "containerMap: removing container" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy"
Nov 02 09:55:37.901233 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:37.901232    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy"
Nov 02 09:55:37.901413 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:37.901294    4449 cpu_manager.go:414] "containerMap: removing container" podUID="230a22f3-5692-4bf8-8897-850431c806ec" containerName="revoke-keys"
Nov 02 09:55:37.901413 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:37.901303    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230a22f3-5692-4bf8-8897-850431c806ec" containerName="revoke-keys"
Nov 02 09:55:37.901519 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:37.901504    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="230a22f3-5692-4bf8-8897-850431c806ec" containerName="revoke-keys"
Nov 02 09:55:37.901571 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:37.901522    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy"
Nov 02 09:55:37.901571 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:37.901538    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy"
Nov 02 09:55:37.922780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:37.922745    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:37.935430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:37.935389    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"]
Nov 02 09:55:37.998243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:37.992049    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:37.998243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:37.992186    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.091659 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.091620    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98145c71-b388-4093-83be-cf5f18cf1af0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.109304 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.095357    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98145c71-b388-4093-83be-cf5f18cf1af0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.136392 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.136357    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.148270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.146881    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.148270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.146940    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.148270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.147013    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxfng\" (UniqueName: \"kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-kube-api-access-xxfng\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.148270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.147073    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.273315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.271621    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.273315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.271802    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.273315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.271924    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.273315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.271966    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98145c71-b388-4093-83be-cf5f18cf1af0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.273315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.272029    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98145c71-b388-4093-83be-cf5f18cf1af0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.273315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.272060    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.273315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.272103    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.273315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.272134    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.273315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.272191    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxfng\" (UniqueName: \"kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-kube-api-access-xxfng\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.273315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.273024    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.344244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.314673    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.344244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.319442    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.344244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.328469    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.344244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.332863    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98145c71-b388-4093-83be-cf5f18cf1af0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.385629 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.385547    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.414595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.413458    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98145c71-b388-4093-83be-cf5f18cf1af0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.435382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.435306    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxfng\" (UniqueName: \"kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-kube-api-access-xxfng\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.522778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.522734    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"]
Nov 02 09:55:38.547299 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.524109    4449 cpu_manager.go:414] "containerMap: removing container" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy"
Nov 02 09:55:38.547547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.547520    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f4c5ab-d3da-449e-b5dc-d6fdcf0751cc" containerName="istio-proxy"
Nov 02 09:55:38.578244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.554196    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"
Nov 02 09:55:38.591510 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.591433    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7cnfk\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.633260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.606879    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:55:38.633260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.606949    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:55:38.633260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.607597    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:55:38.633260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.607683    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:55:38.688240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.669702    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"]
Nov 02 09:55:38.718244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.690929    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:38.769310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.726303    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswnf\" (UniqueName: \"kubernetes.io/projected/26636211-ba44-4e02-9407-2f65b0503f72-kube-api-access-jswnf\") pod \"maas-api-revoke-sub-4ced927a-74m87\" (UID: \"26636211-ba44-4e02-9407-2f65b0503f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"
Nov 02 09:55:38.769310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.726375    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ced927a-74m87\" (UID: \"26636211-ba44-4e02-9407-2f65b0503f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"
Nov 02 09:55:38.769310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.726611    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ced927a-74m87\" (UID: \"26636211-ba44-4e02-9407-2f65b0503f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"
Nov 02 09:55:38.927260 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.904151    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswnf\" (UniqueName: \"kubernetes.io/projected/26636211-ba44-4e02-9407-2f65b0503f72-kube-api-access-jswnf\") pod \"maas-api-revoke-sub-4ced927a-74m87\" (UID: \"26636211-ba44-4e02-9407-2f65b0503f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"
Nov 02 09:55:38.980294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.931303    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ced927a-74m87\" (UID: \"26636211-ba44-4e02-9407-2f65b0503f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"
Nov 02 09:55:38.980294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.931648    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ced927a-74m87\" (UID: \"26636211-ba44-4e02-9407-2f65b0503f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"
Nov 02 09:55:38.980294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:38.932717    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ced927a-74m87\" (UID: \"26636211-ba44-4e02-9407-2f65b0503f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"
Nov 02 09:55:39.174057 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:39.173984    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswnf\" (UniqueName: \"kubernetes.io/projected/26636211-ba44-4e02-9407-2f65b0503f72-kube-api-access-jswnf\") pod \"maas-api-revoke-sub-4ced927a-74m87\" (UID: \"26636211-ba44-4e02-9407-2f65b0503f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"
Nov 02 09:55:39.271249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:39.252758    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ced927a-74m87\" (UID: \"26636211-ba44-4e02-9407-2f65b0503f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"
Nov 02 09:55:39.295245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:39.254332    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:39.295245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:39.280039    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 09:55:39.340845 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:39.338478    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n" podUID="7968e57e-6e72-4ef2-af9f-82c111d870f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:39.401676 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:39.390964    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"
Nov 02 09:55:39.536243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:39.532025    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 02 09:55:39.607974 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:39.607600    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:40.278854 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.278800    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.156:8000/ready\": context deadline exceeded"
Nov 02 09:55:40.279998 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.279966    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:55:40.318868 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.318836    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:55:40.380134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.379680    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:55:40.380134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.379774    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:55:40.380134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.379818    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:55:40.387722 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.387696    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"]
Nov 02 09:55:40.586995 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:55:40.586965    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26636211_ba44_4e02_9407_2f65b0503f72.slice/crio-c0fb87915e96d24d4bb13a7102c962af6baf015de77c9021b8f1da2e5f2dc32f WatchSource:0}: Error finding container c0fb87915e96d24d4bb13a7102c962af6baf015de77c9021b8f1da2e5f2dc32f: Status 404 returned error can't find the container with id c0fb87915e96d24d4bb13a7102c962af6baf015de77c9021b8f1da2e5f2dc32f
Nov 02 09:55:40.591247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.591198    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"]
Nov 02 09:55:40.704613 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.704571    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"]
Nov 02 09:55:40.735241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.733178    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"]
Nov 02 09:55:40.735241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.733351    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"
Nov 02 09:55:40.793132 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.793101    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7458c4da-rjbr6\" (UID: \"4560961e-36b1-4312-96d2-371f0e228c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"
Nov 02 09:55:40.793349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.793295    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9z7\" (UniqueName: \"kubernetes.io/projected/4560961e-36b1-4312-96d2-371f0e228c73-kube-api-access-2c9z7\") pod \"maas-api-revoke-sub-7458c4da-rjbr6\" (UID: \"4560961e-36b1-4312-96d2-371f0e228c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"
Nov 02 09:55:40.793349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.793338    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7458c4da-rjbr6\" (UID: \"4560961e-36b1-4312-96d2-371f0e228c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"
Nov 02 09:55:40.895014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.894925    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7458c4da-rjbr6\" (UID: \"4560961e-36b1-4312-96d2-371f0e228c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"
Nov 02 09:55:40.895180 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.895026    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9z7\" (UniqueName: \"kubernetes.io/projected/4560961e-36b1-4312-96d2-371f0e228c73-kube-api-access-2c9z7\") pod \"maas-api-revoke-sub-7458c4da-rjbr6\" (UID: \"4560961e-36b1-4312-96d2-371f0e228c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"
Nov 02 09:55:40.895180 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.895049    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7458c4da-rjbr6\" (UID: \"4560961e-36b1-4312-96d2-371f0e228c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"
Nov 02 09:55:40.895831 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.895798    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7458c4da-rjbr6\" (UID: \"4560961e-36b1-4312-96d2-371f0e228c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"
Nov 02 09:55:40.897821 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.897792    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7458c4da-rjbr6\" (UID: \"4560961e-36b1-4312-96d2-371f0e228c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"
Nov 02 09:55:40.907856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:40.907820    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9z7\" (UniqueName: \"kubernetes.io/projected/4560961e-36b1-4312-96d2-371f0e228c73-kube-api-access-2c9z7\") pod \"maas-api-revoke-sub-7458c4da-rjbr6\" (UID: \"4560961e-36b1-4312-96d2-371f0e228c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"
Nov 02 09:55:41.054814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:41.054777    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"
Nov 02 09:55:41.176822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:41.176703    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87" event={"ID":"26636211-ba44-4e02-9407-2f65b0503f72","Type":"ContainerStarted","Data":"327c01f437500bd987058811f0fac954fd7103dd10f4b5901ecad361d6f23a20"}
Nov 02 09:55:41.176822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:41.176789    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87" event={"ID":"26636211-ba44-4e02-9407-2f65b0503f72","Type":"ContainerStarted","Data":"c0fb87915e96d24d4bb13a7102c962af6baf015de77c9021b8f1da2e5f2dc32f"}
Nov 02 09:55:41.179130 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:41.179096    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" event={"ID":"98145c71-b388-4093-83be-cf5f18cf1af0","Type":"ContainerStarted","Data":"ba9e4e41a821070bd7eac46538180d5ed5740a4de6b3ff439766252b1a1682d4"}
Nov 02 09:55:41.179297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:41.179156    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" event={"ID":"98145c71-b388-4093-83be-cf5f18cf1af0","Type":"ContainerStarted","Data":"c5fe06a2d1783679db3a80c8af3c18d4af62aadfc7bde3de69a31e056ac0c15f"}
Nov 02 09:55:41.203272 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:41.203186    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87" podStartSLOduration=3.20316613 podStartE2EDuration="3.20316613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:41.195392806 +0000 UTC m=+1218.133679609" watchObservedRunningTime="2026-11-02 09:55:41.20316613 +0000 UTC m=+1218.141452907"
Nov 02 09:55:41.220970 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:41.220914    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podStartSLOduration=4.220898053 podStartE2EDuration="4.220898053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:41.220162699 +0000 UTC m=+1218.158449495" watchObservedRunningTime="2026-11-02 09:55:41.220898053 +0000 UTC m=+1218.159184831"
Nov 02 09:55:41.353561 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:55:41.353521    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4560961e_36b1_4312_96d2_371f0e228c73.slice/crio-ca5e89887e79d46393c9caafebc6c275aaccc2cfd174a7b8c2df05b4be1d9bb1 WatchSource:0}: Error finding container ca5e89887e79d46393c9caafebc6c275aaccc2cfd174a7b8c2df05b4be1d9bb1: Status 404 returned error can't find the container with id ca5e89887e79d46393c9caafebc6c275aaccc2cfd174a7b8c2df05b4be1d9bb1
Nov 02 09:55:41.354561 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:41.354463    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"]
Nov 02 09:55:41.726437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:41.726343    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:41.727903 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:41.727870    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 02 09:55:41.728065 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:41.727931    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 02 09:55:42.191067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:42.191033    4449 generic.go:356] "Generic (PLEG): container finished" podID="26636211-ba44-4e02-9407-2f65b0503f72" containerID="327c01f437500bd987058811f0fac954fd7103dd10f4b5901ecad361d6f23a20" exitCode=0
Nov 02 09:55:42.191296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:42.191117    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87" event={"ID":"26636211-ba44-4e02-9407-2f65b0503f72","Type":"ContainerDied","Data":"327c01f437500bd987058811f0fac954fd7103dd10f4b5901ecad361d6f23a20"}
Nov 02 09:55:42.194326 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:42.194298    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6" event={"ID":"4560961e-36b1-4312-96d2-371f0e228c73","Type":"ContainerStarted","Data":"64c5b65c1481a3aea072e7933b53392f8cabf760aab8e473f38d49d1bdedbbdb"}
Nov 02 09:55:42.194443 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:42.194341    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6" event={"ID":"4560961e-36b1-4312-96d2-371f0e228c73","Type":"ContainerStarted","Data":"ca5e89887e79d46393c9caafebc6c275aaccc2cfd174a7b8c2df05b4be1d9bb1"}
Nov 02 09:55:42.228148 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:42.228089    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6" podStartSLOduration=2.228074845 podStartE2EDuration="2.228074845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:42.226754852 +0000 UTC m=+1219.165041619" watchObservedRunningTime="2026-11-02 09:55:42.228074845 +0000 UTC m=+1219.166361620"
Nov 02 09:55:42.726114 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:42.726082    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 02 09:55:42.726489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:42.726136    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 02 09:55:43.206583 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.206545    4449 generic.go:356] "Generic (PLEG): container finished" podID="4560961e-36b1-4312-96d2-371f0e228c73" containerID="64c5b65c1481a3aea072e7933b53392f8cabf760aab8e473f38d49d1bdedbbdb" exitCode=0
Nov 02 09:55:43.211617 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.211584    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6" event={"ID":"4560961e-36b1-4312-96d2-371f0e228c73","Type":"ContainerDied","Data":"64c5b65c1481a3aea072e7933b53392f8cabf760aab8e473f38d49d1bdedbbdb"}
Nov 02 09:55:43.486828 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.486756    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"
Nov 02 09:55:43.543654 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.543616    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-openshift-ca\") pod \"26636211-ba44-4e02-9407-2f65b0503f72\" (UID: \"26636211-ba44-4e02-9407-2f65b0503f72\") "
Nov 02 09:55:43.543833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.543714    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26636211-ba44-4e02-9407-2f65b0503f72-kube-api-access-jswnf\" (UniqueName: \"kubernetes.io/projected/26636211-ba44-4e02-9407-2f65b0503f72-kube-api-access-jswnf\") pod \"26636211-ba44-4e02-9407-2f65b0503f72\" (UID: \"26636211-ba44-4e02-9407-2f65b0503f72\") "
Nov 02 09:55:43.543882 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.543845    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-xks-ca\") pod \"26636211-ba44-4e02-9407-2f65b0503f72\" (UID: \"26636211-ba44-4e02-9407-2f65b0503f72\") "
Nov 02 09:55:43.544056 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.544031    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-openshift-ca" pod "26636211-ba44-4e02-9407-2f65b0503f72" (UID: "26636211-ba44-4e02-9407-2f65b0503f72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:43.544545 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.544516    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:43.546762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.546676    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-xks-ca" pod "26636211-ba44-4e02-9407-2f65b0503f72" (UID: "26636211-ba44-4e02-9407-2f65b0503f72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:55:43.546865 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.546776    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26636211-ba44-4e02-9407-2f65b0503f72-kube-api-access-jswnf" pod "26636211-ba44-4e02-9407-2f65b0503f72" (UID: "26636211-ba44-4e02-9407-2f65b0503f72"). InnerVolumeSpecName "kube-api-access-jswnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:43.645648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.645607    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jswnf\" (UniqueName: \"kubernetes.io/projected/26636211-ba44-4e02-9407-2f65b0503f72-kube-api-access-jswnf\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:43.645648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.645642    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26636211-ba44-4e02-9407-2f65b0503f72-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:43.726036 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.726001    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 02 09:55:43.726206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:43.726061    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 02 09:55:44.217453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.217425    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"
Nov 02 09:55:44.217647 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.217455    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87" event={"ID":"26636211-ba44-4e02-9407-2f65b0503f72","Type":"ContainerDied","Data":"c0fb87915e96d24d4bb13a7102c962af6baf015de77c9021b8f1da2e5f2dc32f"}
Nov 02 09:55:44.217647 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.217514    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0fb87915e96d24d4bb13a7102c962af6baf015de77c9021b8f1da2e5f2dc32f"
Nov 02 09:55:44.471301 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.459440    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"
Nov 02 09:55:44.590795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.590753    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-xks-ca\") pod \"4560961e-36b1-4312-96d2-371f0e228c73\" (UID: \"4560961e-36b1-4312-96d2-371f0e228c73\") "
Nov 02 09:55:44.591178 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.591157    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4560961e-36b1-4312-96d2-371f0e228c73-kube-api-access-2c9z7\" (UniqueName: \"kubernetes.io/projected/4560961e-36b1-4312-96d2-371f0e228c73-kube-api-access-2c9z7\") pod \"4560961e-36b1-4312-96d2-371f0e228c73\" (UID: \"4560961e-36b1-4312-96d2-371f0e228c73\") "
Nov 02 09:55:44.591458 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.591437    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-openshift-ca\") pod \"4560961e-36b1-4312-96d2-371f0e228c73\" (UID: \"4560961e-36b1-4312-96d2-371f0e228c73\") "
Nov 02 09:55:44.591834 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.591811    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-openshift-ca" pod "4560961e-36b1-4312-96d2-371f0e228c73" (UID: "4560961e-36b1-4312-96d2-371f0e228c73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:44.592508 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.592483    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:44.593419 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.593378    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-xks-ca" pod "4560961e-36b1-4312-96d2-371f0e228c73" (UID: "4560961e-36b1-4312-96d2-371f0e228c73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:55:44.593861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.593834    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4560961e-36b1-4312-96d2-371f0e228c73-kube-api-access-2c9z7" pod "4560961e-36b1-4312-96d2-371f0e228c73" (UID: "4560961e-36b1-4312-96d2-371f0e228c73"). InnerVolumeSpecName "kube-api-access-2c9z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:44.694056 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.694002    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4560961e-36b1-4312-96d2-371f0e228c73-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:44.694056 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.694042    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c9z7\" (UniqueName: \"kubernetes.io/projected/4560961e-36b1-4312-96d2-371f0e228c73-kube-api-access-2c9z7\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:44.726032 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.725945    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 02 09:55:44.726032 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:44.726002    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 02 09:55:45.027838 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:45.027733    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"]
Nov 02 09:55:45.033785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:45.033750    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0de25fa-jbpcm"]
Nov 02 09:55:45.209018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:45.208982    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25edb50b-6bac-4c2d-9045-f1be89706a0b" path="/var/lib/kubelet/pods/25edb50b-6bac-4c2d-9045-f1be89706a0b/volumes"
Nov 02 09:55:45.209605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:45.209588    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"]
Nov 02 09:55:45.211149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:45.211122    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"]
Nov 02 09:55:45.240446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:45.240397    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6" event={"ID":"4560961e-36b1-4312-96d2-371f0e228c73","Type":"ContainerDied","Data":"ca5e89887e79d46393c9caafebc6c275aaccc2cfd174a7b8c2df05b4be1d9bb1"}
Nov 02 09:55:45.240700 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:45.240669    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca5e89887e79d46393c9caafebc6c275aaccc2cfd174a7b8c2df05b4be1d9bb1"
Nov 02 09:55:45.240839 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:45.240793    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"
Nov 02 09:55:45.726396 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:45.726306    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 02 09:55:45.726589 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:45.726459    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 02 09:55:46.726980 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:46.726708    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 02 09:55:46.726980 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:46.726758    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 02 09:55:47.726060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:47.726019    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 02 09:55:47.726268 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:47.726090    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 02 09:55:48.026902 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.026807    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"]
Nov 02 09:55:48.033307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.033264    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0004c2-kn2lx"]
Nov 02 09:55:48.403449 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.403418    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"]
Nov 02 09:55:48.404143 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.404118    4449 cpu_manager.go:414] "containerMap: removing container" podUID="26636211-ba44-4e02-9407-2f65b0503f72" containerName="revoke-keys"
Nov 02 09:55:48.404143 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.404135    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26636211-ba44-4e02-9407-2f65b0503f72" containerName="revoke-keys"
Nov 02 09:55:48.404287 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.404168    4449 cpu_manager.go:414] "containerMap: removing container" podUID="4560961e-36b1-4312-96d2-371f0e228c73" containerName="revoke-keys"
Nov 02 09:55:48.404287 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.404175    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4560961e-36b1-4312-96d2-371f0e228c73" containerName="revoke-keys"
Nov 02 09:55:48.404362 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.404327    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="4560961e-36b1-4312-96d2-371f0e228c73" containerName="revoke-keys"
Nov 02 09:55:48.404362 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.404343    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="26636211-ba44-4e02-9407-2f65b0503f72" containerName="revoke-keys"
Nov 02 09:55:48.409199 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.409181    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"
Nov 02 09:55:48.415382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.415360    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:55:48.415659 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.415571    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:55:48.415659 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.415592    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:55:48.415659 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.415629    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:55:48.416951 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.416928    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"]
Nov 02 09:55:48.470396 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.470349    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2af7639a-8dzf2\" (UID: \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"
Nov 02 09:55:48.470558 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.470537    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2af7639a-8dzf2\" (UID: \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"
Nov 02 09:55:48.470602 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.470573    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxp2j\" (UniqueName: \"kubernetes.io/projected/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-kube-api-access-xxp2j\") pod \"maas-api-revoke-sub-2af7639a-8dzf2\" (UID: \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"
Nov 02 09:55:48.572733 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.572688    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2af7639a-8dzf2\" (UID: \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"
Nov 02 09:55:48.572943 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.572813    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2af7639a-8dzf2\" (UID: \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"
Nov 02 09:55:48.572943 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.572842    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxp2j\" (UniqueName: \"kubernetes.io/projected/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-kube-api-access-xxp2j\") pod \"maas-api-revoke-sub-2af7639a-8dzf2\" (UID: \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"
Nov 02 09:55:48.573417 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.573395    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2af7639a-8dzf2\" (UID: \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"
Nov 02 09:55:48.575307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.575286    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2af7639a-8dzf2\" (UID: \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"
Nov 02 09:55:48.585868 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.585830    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxp2j\" (UniqueName: \"kubernetes.io/projected/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-kube-api-access-xxp2j\") pod \"maas-api-revoke-sub-2af7639a-8dzf2\" (UID: \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"
Nov 02 09:55:48.739276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.728327    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 02 09:55:48.739276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.728430    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 02 09:55:48.739276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:48.736404    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"
Nov 02 09:55:49.092036 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:55:49.092005    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3157e9f7_d8a6_4ce2_81e1_07337f6d9bd0.slice/crio-a95efa00a04a07d499cf5ad0d3127f0ef7aaeec0d30794dd52420ca458ddbf29 WatchSource:0}: Error finding container a95efa00a04a07d499cf5ad0d3127f0ef7aaeec0d30794dd52420ca458ddbf29: Status 404 returned error can't find the container with id a95efa00a04a07d499cf5ad0d3127f0ef7aaeec0d30794dd52420ca458ddbf29
Nov 02 09:55:49.092482 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:49.092431    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"]
Nov 02 09:55:49.209323 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:49.209284    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981a9928-8e77-4f76-96a1-59f1e541c65e" path="/var/lib/kubelet/pods/981a9928-8e77-4f76-96a1-59f1e541c65e/volumes"
Nov 02 09:55:49.285401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:49.285369    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2" event={"ID":"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0","Type":"ContainerStarted","Data":"a95efa00a04a07d499cf5ad0d3127f0ef7aaeec0d30794dd52420ca458ddbf29"}
Nov 02 09:55:50.210460 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.210425    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf" podUID="98dd9543-5592-461c-94e8-7fffe7079d7f" containerName="payload-processing" containerID="cri-o://057dc6eac0ce8a0776dba08b7c4026d9aeb21c8037af7749952fae6aaf59aac8" gracePeriod=25
Nov 02 09:55:50.211979 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.211945    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm" podUID="11a79a51-82c7-4a50-b8aa-6a155be721e3" containerName="payload-pre-processing" containerID="cri-o://39a141d5dc02c5deb5d456585a28bdd625042866fa971905e4d988b33863665a" gracePeriod=25
Nov 02 09:55:50.302546 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.302516    4449 generic.go:356] "Generic (PLEG): container finished" podID="3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0" containerID="09ab49d976642b9d4547bba9f8a89a06b42b8da1e7e9b7f89e772647ed2b56aa" exitCode=0
Nov 02 09:55:50.302716 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.302602    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2" event={"ID":"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0","Type":"ContainerDied","Data":"09ab49d976642b9d4547bba9f8a89a06b42b8da1e7e9b7f89e772647ed2b56aa"}
Nov 02 09:55:50.566737 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.565938    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"
Nov 02 09:55:50.579704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.579672    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"
Nov 02 09:55:50.712187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.711016    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98dd9543-5592-461c-94e8-7fffe7079d7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98dd9543-5592-461c-94e8-7fffe7079d7f-plugins-config-volume\") pod \"98dd9543-5592-461c-94e8-7fffe7079d7f\" (UID: \"98dd9543-5592-461c-94e8-7fffe7079d7f\") "
Nov 02 09:55:50.712187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.711234    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a79a51-82c7-4a50-b8aa-6a155be721e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a79a51-82c7-4a50-b8aa-6a155be721e3-plugins-config-volume\") pod \"11a79a51-82c7-4a50-b8aa-6a155be721e3\" (UID: \"11a79a51-82c7-4a50-b8aa-6a155be721e3\") "
Nov 02 09:55:50.712187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.711443    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a79a51-82c7-4a50-b8aa-6a155be721e3-kube-api-access-tq4dk\" (UniqueName: \"kubernetes.io/projected/11a79a51-82c7-4a50-b8aa-6a155be721e3-kube-api-access-tq4dk\") pod \"11a79a51-82c7-4a50-b8aa-6a155be721e3\" (UID: \"11a79a51-82c7-4a50-b8aa-6a155be721e3\") "
Nov 02 09:55:50.712187 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.711552    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98dd9543-5592-461c-94e8-7fffe7079d7f-kube-api-access-kqfjk\" (UniqueName: \"kubernetes.io/projected/98dd9543-5592-461c-94e8-7fffe7079d7f-kube-api-access-kqfjk\") pod \"98dd9543-5592-461c-94e8-7fffe7079d7f\" (UID: \"98dd9543-5592-461c-94e8-7fffe7079d7f\") "
Nov 02 09:55:50.712736 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.712692    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a79a51-82c7-4a50-b8aa-6a155be721e3-plugins-config-volume" pod "11a79a51-82c7-4a50-b8aa-6a155be721e3" (UID: "11a79a51-82c7-4a50-b8aa-6a155be721e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:50.713500 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.713469    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98dd9543-5592-461c-94e8-7fffe7079d7f-plugins-config-volume" pod "98dd9543-5592-461c-94e8-7fffe7079d7f" (UID: "98dd9543-5592-461c-94e8-7fffe7079d7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:50.715401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.715370    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a79a51-82c7-4a50-b8aa-6a155be721e3-kube-api-access-tq4dk" pod "11a79a51-82c7-4a50-b8aa-6a155be721e3" (UID: "11a79a51-82c7-4a50-b8aa-6a155be721e3"). InnerVolumeSpecName "kube-api-access-tq4dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:50.720016 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.719980    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98dd9543-5592-461c-94e8-7fffe7079d7f-kube-api-access-kqfjk" pod "98dd9543-5592-461c-94e8-7fffe7079d7f" (UID: "98dd9543-5592-461c-94e8-7fffe7079d7f"). InnerVolumeSpecName "kube-api-access-kqfjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:50.727147 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.727111    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:55:50.727312 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.727171    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:50.765816 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.765780    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:50.767433 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.767401    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:50.767603 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.767527    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 09:55:50.814425 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.814320    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98dd9543-5592-461c-94e8-7fffe7079d7f-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:50.814425 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.814358    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a79a51-82c7-4a50-b8aa-6a155be721e3-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:50.814425 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.814373    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq4dk\" (UniqueName: \"kubernetes.io/projected/11a79a51-82c7-4a50-b8aa-6a155be721e3-kube-api-access-tq4dk\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:50.814425 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:50.814387    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqfjk\" (UniqueName: \"kubernetes.io/projected/98dd9543-5592-461c-94e8-7fffe7079d7f-kube-api-access-kqfjk\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:51.314211 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.314172    4449 generic.go:356] "Generic (PLEG): container finished" podID="98dd9543-5592-461c-94e8-7fffe7079d7f" containerID="057dc6eac0ce8a0776dba08b7c4026d9aeb21c8037af7749952fae6aaf59aac8" exitCode=0
Nov 02 09:55:51.314693 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.314246    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf" event={"ID":"98dd9543-5592-461c-94e8-7fffe7079d7f","Type":"ContainerDied","Data":"057dc6eac0ce8a0776dba08b7c4026d9aeb21c8037af7749952fae6aaf59aac8"}
Nov 02 09:55:51.314693 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.314261    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"
Nov 02 09:55:51.314693 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.314317    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf" event={"ID":"98dd9543-5592-461c-94e8-7fffe7079d7f","Type":"ContainerDied","Data":"395a2fec7982f2a9235d4039be7a1f6ee52b4bd2c96ce0bd02ad896a434000c4"}
Nov 02 09:55:51.314693 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.314342    4449 scope.go:122] "RemoveContainer" containerID="057dc6eac0ce8a0776dba08b7c4026d9aeb21c8037af7749952fae6aaf59aac8"
Nov 02 09:55:51.316277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.316205    4449 generic.go:356] "Generic (PLEG): container finished" podID="11a79a51-82c7-4a50-b8aa-6a155be721e3" containerID="39a141d5dc02c5deb5d456585a28bdd625042866fa971905e4d988b33863665a" exitCode=0
Nov 02 09:55:51.316409 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.316271    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm" event={"ID":"11a79a51-82c7-4a50-b8aa-6a155be721e3","Type":"ContainerDied","Data":"39a141d5dc02c5deb5d456585a28bdd625042866fa971905e4d988b33863665a"}
Nov 02 09:55:51.316409 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.316331    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm" event={"ID":"11a79a51-82c7-4a50-b8aa-6a155be721e3","Type":"ContainerDied","Data":"8a97d14a25935233484d616865c9499d7d7d45701dc19d194ceb77c0b5122014"}
Nov 02 09:55:51.316409 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.316280    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"
Nov 02 09:55:51.330788 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.330764    4449 scope.go:122] "RemoveContainer" containerID="057dc6eac0ce8a0776dba08b7c4026d9aeb21c8037af7749952fae6aaf59aac8"
Nov 02 09:55:51.331173 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:55:51.331150    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057dc6eac0ce8a0776dba08b7c4026d9aeb21c8037af7749952fae6aaf59aac8\": container with ID starting with 057dc6eac0ce8a0776dba08b7c4026d9aeb21c8037af7749952fae6aaf59aac8 not found: ID does not exist" containerID="057dc6eac0ce8a0776dba08b7c4026d9aeb21c8037af7749952fae6aaf59aac8"
Nov 02 09:55:51.331264 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.331183    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057dc6eac0ce8a0776dba08b7c4026d9aeb21c8037af7749952fae6aaf59aac8"} err="failed to get container status \"057dc6eac0ce8a0776dba08b7c4026d9aeb21c8037af7749952fae6aaf59aac8\": rpc error: code = NotFound desc = could not find container \"057dc6eac0ce8a0776dba08b7c4026d9aeb21c8037af7749952fae6aaf59aac8\": container with ID starting with 057dc6eac0ce8a0776dba08b7c4026d9aeb21c8037af7749952fae6aaf59aac8 not found: ID does not exist"
Nov 02 09:55:51.331264 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.331203    4449 scope.go:122] "RemoveContainer" containerID="39a141d5dc02c5deb5d456585a28bdd625042866fa971905e4d988b33863665a"
Nov 02 09:55:51.343274 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.343003    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"]
Nov 02 09:55:51.346617 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.346591    4449 scope.go:122] "RemoveContainer" containerID="39a141d5dc02c5deb5d456585a28bdd625042866fa971905e4d988b33863665a"
Nov 02 09:55:51.346993 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:55:51.346966    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a141d5dc02c5deb5d456585a28bdd625042866fa971905e4d988b33863665a\": container with ID starting with 39a141d5dc02c5deb5d456585a28bdd625042866fa971905e4d988b33863665a not found: ID does not exist" containerID="39a141d5dc02c5deb5d456585a28bdd625042866fa971905e4d988b33863665a"
Nov 02 09:55:51.347094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.346999    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a141d5dc02c5deb5d456585a28bdd625042866fa971905e4d988b33863665a"} err="failed to get container status \"39a141d5dc02c5deb5d456585a28bdd625042866fa971905e4d988b33863665a\": rpc error: code = NotFound desc = could not find container \"39a141d5dc02c5deb5d456585a28bdd625042866fa971905e4d988b33863665a\": container with ID starting with 39a141d5dc02c5deb5d456585a28bdd625042866fa971905e4d988b33863665a not found: ID does not exist"
Nov 02 09:55:51.349528 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.349502    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a71884-7f8968dfdf-72wsf"]
Nov 02 09:55:51.359249 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.359198    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"]
Nov 02 09:55:51.364946 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.364918    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a71884-94f55f9dc-pzflm"]
Nov 02 09:55:51.519208 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.519177    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"
Nov 02 09:55:51.654266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.628713    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-openshift-ca\") pod \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\" (UID: \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\") "
Nov 02 09:55:51.654266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.628796    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-kube-api-access-xxp2j\" (UniqueName: \"kubernetes.io/projected/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-kube-api-access-xxp2j\") pod \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\" (UID: \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\") "
Nov 02 09:55:51.654266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.628950    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-xks-ca\") pod \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\" (UID: \"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0\") "
Nov 02 09:55:51.704242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.692490    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-openshift-ca" pod "3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0" (UID: "3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:51.704242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.696743    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-kube-api-access-xxp2j" pod "3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0" (UID: "3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0"). InnerVolumeSpecName "kube-api-access-xxp2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:51.733360 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.731934    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:51.733360 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.731979    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxp2j\" (UniqueName: \"kubernetes.io/projected/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-kube-api-access-xxp2j\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:51.733732 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.733687    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-xks-ca" pod "3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0" (UID: "3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:55:51.833574 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.833533    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:51.961532 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:51.954160    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"]
Nov 02 09:55:52.344238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:52.338245    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"
Nov 02 09:55:52.344238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:52.339660    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2" event={"ID":"3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0","Type":"ContainerDied","Data":"a95efa00a04a07d499cf5ad0d3127f0ef7aaeec0d30794dd52420ca458ddbf29"}
Nov 02 09:55:52.344238 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:52.339716    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a95efa00a04a07d499cf5ad0d3127f0ef7aaeec0d30794dd52420ca458ddbf29"
Nov 02 09:55:53.216085 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:53.216050    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a79a51-82c7-4a50-b8aa-6a155be721e3" path="/var/lib/kubelet/pods/11a79a51-82c7-4a50-b8aa-6a155be721e3/volumes"
Nov 02 09:55:53.216687 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:53.216651    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98dd9543-5592-461c-94e8-7fffe7079d7f" path="/var/lib/kubelet/pods/98dd9543-5592-461c-94e8-7fffe7079d7f/volumes"
Nov 02 09:55:56.511343 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:56.511293    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 02 09:55:57.082245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:57.081281    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8" podUID="33a2bffe-6868-4ac6-9e24-e59d84441b77" containerName="payload-processing" containerID="cri-o://a3fab60121be25f772a57cbc423666e0230883b7066ac62ddb3d9334b354871a" gracePeriod=25
Nov 02 09:55:57.574148 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:57.573361    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:57.732494 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:57.731994    4449 generic.go:356] "Generic (PLEG): container finished" podID="33a2bffe-6868-4ac6-9e24-e59d84441b77" containerID="a3fab60121be25f772a57cbc423666e0230883b7066ac62ddb3d9334b354871a" exitCode=0
Nov 02 09:55:57.732494 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:57.732392    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8" event={"ID":"33a2bffe-6868-4ac6-9e24-e59d84441b77","Type":"ContainerDied","Data":"a3fab60121be25f772a57cbc423666e0230883b7066ac62ddb3d9334b354871a"}
Nov 02 09:55:58.493195 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:58.492285    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"]
Nov 02 09:55:58.577244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:58.576538    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:58.701253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:58.597078    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:55:58.921348 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:58.921290    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:55:58.921904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:58.921868    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:55:58.922020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:58.921933    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:58.951136 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:58.951103    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:55:59.025423 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.025336    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-routing-overlay-volume\") pod \"33a2bffe-6868-4ac6-9e24-e59d84441b77\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") "
Nov 02 09:55:59.025869 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.025839    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-provider-credentials\" (UniqueName: \"kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-provider-credentials\") pod \"33a2bffe-6868-4ac6-9e24-e59d84441b77\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") "
Nov 02 09:55:59.026298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.026276    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-kube-api-access-7fwf7\" (UniqueName: \"kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-kube-api-access-7fwf7\") pod \"33a2bffe-6868-4ac6-9e24-e59d84441b77\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") "
Nov 02 09:55:59.026490 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.025902    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-routing-overlay-volume" pod "33a2bffe-6868-4ac6-9e24-e59d84441b77" (UID: "33a2bffe-6868-4ac6-9e24-e59d84441b77"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:59.026622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.026602    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-plugins-config-volume\") pod \"33a2bffe-6868-4ac6-9e24-e59d84441b77\" (UID: \"33a2bffe-6868-4ac6-9e24-e59d84441b77\") "
Nov 02 09:55:59.027801 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.027778    4449 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-routing-overlay-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:59.033703 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.033680    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66aab2e-m8zcz"]
Nov 02 09:55:59.041577 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.040388    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-plugins-config-volume" pod "33a2bffe-6868-4ac6-9e24-e59d84441b77" (UID: "33a2bffe-6868-4ac6-9e24-e59d84441b77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:59.046693 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.046477    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-kube-api-access-7fwf7" pod "33a2bffe-6868-4ac6-9e24-e59d84441b77" (UID: "33a2bffe-6868-4ac6-9e24-e59d84441b77"). InnerVolumeSpecName "kube-api-access-7fwf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:59.068814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.068777    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-provider-credentials" pod "33a2bffe-6868-4ac6-9e24-e59d84441b77" (UID: "33a2bffe-6868-4ac6-9e24-e59d84441b77"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:59.090243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.078084    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"
Nov 02 09:55:59.098453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.096457    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8" event={"ID":"33a2bffe-6868-4ac6-9e24-e59d84441b77","Type":"ContainerDied","Data":"a4c1d077e7637c030019ad6d4c63d7815927750a0b6c113164ca40c1d345e7df"}
Nov 02 09:55:59.098453 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.096540    4449 scope.go:122] "RemoveContainer" containerID="a3fab60121be25f772a57cbc423666e0230883b7066ac62ddb3d9334b354871a"
Nov 02 09:55:59.113999 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.113974    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"]
Nov 02 09:55:59.115514 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.115481    4449 cpu_manager.go:414] "containerMap: removing container" podUID="98dd9543-5592-461c-94e8-7fffe7079d7f" containerName="payload-processing"
Nov 02 09:55:59.115687 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.115674    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98dd9543-5592-461c-94e8-7fffe7079d7f" containerName="payload-processing"
Nov 02 09:55:59.115802 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.115791    4449 cpu_manager.go:414] "containerMap: removing container" podUID="11a79a51-82c7-4a50-b8aa-6a155be721e3" containerName="payload-pre-processing"
Nov 02 09:55:59.115888 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.115865    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a79a51-82c7-4a50-b8aa-6a155be721e3" containerName="payload-pre-processing"
Nov 02 09:55:59.116011 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.116000    4449 cpu_manager.go:414] "containerMap: removing container" podUID="33a2bffe-6868-4ac6-9e24-e59d84441b77" containerName="payload-processing"
Nov 02 09:55:59.116130 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.116115    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a2bffe-6868-4ac6-9e24-e59d84441b77" containerName="payload-processing"
Nov 02 09:55:59.116259 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.116248    4449 cpu_manager.go:414] "containerMap: removing container" podUID="3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0" containerName="revoke-keys"
Nov 02 09:55:59.116348 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.116339    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0" containerName="revoke-keys"
Nov 02 09:55:59.116736 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.116600    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a79a51-82c7-4a50-b8aa-6a155be721e3" containerName="payload-pre-processing"
Nov 02 09:55:59.116736 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.116628    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a2bffe-6868-4ac6-9e24-e59d84441b77" containerName="payload-processing"
Nov 02 09:55:59.116736 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.116642    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0" containerName="revoke-keys"
Nov 02 09:55:59.116736 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.116658    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="98dd9543-5592-461c-94e8-7fffe7079d7f" containerName="payload-processing"
Nov 02 09:55:59.126997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.126963    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"]
Nov 02 09:55:59.127160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.127086    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"
Nov 02 09:55:59.129547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.129520    4449 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-provider-credentials\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:59.129547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.129552    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fwf7\" (UniqueName: \"kubernetes.io/projected/33a2bffe-6868-4ac6-9e24-e59d84441b77-kube-api-access-7fwf7\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:59.129547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.129569    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a2bffe-6868-4ac6-9e24-e59d84441b77-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:59.135554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.134968    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:55:59.135554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.135034    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:55:59.135554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.135118    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:55:59.135554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.135203    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:55:59.147739 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.147706    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"]
Nov 02 09:55:59.156699 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.156436    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7bdckntz8"]
Nov 02 09:55:59.211159 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.211123    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a2bffe-6868-4ac6-9e24-e59d84441b77" path="/var/lib/kubelet/pods/33a2bffe-6868-4ac6-9e24-e59d84441b77/volumes"
Nov 02 09:55:59.211776 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.211753    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0c263d-3489-4d88-b151-7746d205493e" path="/var/lib/kubelet/pods/da0c263d-3489-4d88-b151-7746d205493e/volumes"
Nov 02 09:55:59.230817 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.230772    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp6bh\" (UniqueName: \"kubernetes.io/projected/ebbb08be-db62-47c4-8d7b-6eb3a3598438-kube-api-access-cp6bh\") pod \"maas-api-revoke-sub-bcb44f15-l64j4\" (UID: \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"
Nov 02 09:55:59.231027 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.230868    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb44f15-l64j4\" (UID: \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"
Nov 02 09:55:59.231355 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.231301    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb44f15-l64j4\" (UID: \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"
Nov 02 09:55:59.332957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.332929    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp6bh\" (UniqueName: \"kubernetes.io/projected/ebbb08be-db62-47c4-8d7b-6eb3a3598438-kube-api-access-cp6bh\") pod \"maas-api-revoke-sub-bcb44f15-l64j4\" (UID: \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"
Nov 02 09:55:59.333130 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.333008    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb44f15-l64j4\" (UID: \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"
Nov 02 09:55:59.333239 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.333204    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb44f15-l64j4\" (UID: \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"
Nov 02 09:55:59.333910 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.333888    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb44f15-l64j4\" (UID: \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"
Nov 02 09:55:59.335566 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.335546    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb44f15-l64j4\" (UID: \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"
Nov 02 09:55:59.345892 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.345870    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp6bh\" (UniqueName: \"kubernetes.io/projected/ebbb08be-db62-47c4-8d7b-6eb3a3598438-kube-api-access-cp6bh\") pod \"maas-api-revoke-sub-bcb44f15-l64j4\" (UID: \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"
Nov 02 09:55:59.444395 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.444358    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"
Nov 02 09:55:59.795759 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:55:59.795734    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbb08be_db62_47c4_8d7b_6eb3a3598438.slice/crio-042373a604b8640e14709a28d3a5121f75856e48d8e19da1f344438ad0968207 WatchSource:0}: Error finding container 042373a604b8640e14709a28d3a5121f75856e48d8e19da1f344438ad0968207: Status 404 returned error can't find the container with id 042373a604b8640e14709a28d3a5121f75856e48d8e19da1f344438ad0968207
Nov 02 09:55:59.796540 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:55:59.796512    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"]
Nov 02 09:56:00.098413 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:00.098377    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4" event={"ID":"ebbb08be-db62-47c4-8d7b-6eb3a3598438","Type":"ContainerStarted","Data":"042373a604b8640e14709a28d3a5121f75856e48d8e19da1f344438ad0968207"}
Nov 02 09:56:01.112695 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:01.112659    4449 generic.go:356] "Generic (PLEG): container finished" podID="ebbb08be-db62-47c4-8d7b-6eb3a3598438" containerID="d6110df14fabb78a03a2be33a21423a8d190b96f58ee8a378dceda39c8d68543" exitCode=0
Nov 02 09:56:01.113115 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:01.112698    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4" event={"ID":"ebbb08be-db62-47c4-8d7b-6eb3a3598438","Type":"ContainerDied","Data":"d6110df14fabb78a03a2be33a21423a8d190b96f58ee8a378dceda39c8d68543"}
Nov 02 09:56:02.030053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:02.030000    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"]
Nov 02 09:56:02.036691 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:02.036652    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d493b49-c8fh4"]
Nov 02 09:56:02.325198 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:02.325163    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"
Nov 02 09:56:02.422345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:02.422309    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-xks-ca\") pod \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\" (UID: \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\") "
Nov 02 09:56:02.422538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:02.422369    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbb08be-db62-47c4-8d7b-6eb3a3598438-kube-api-access-cp6bh\" (UniqueName: \"kubernetes.io/projected/ebbb08be-db62-47c4-8d7b-6eb3a3598438-kube-api-access-cp6bh\") pod \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\" (UID: \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\") "
Nov 02 09:56:02.422538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:02.422395    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-openshift-ca\") pod \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\" (UID: \"ebbb08be-db62-47c4-8d7b-6eb3a3598438\") "
Nov 02 09:56:02.422909 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:02.422878    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-openshift-ca" pod "ebbb08be-db62-47c4-8d7b-6eb3a3598438" (UID: "ebbb08be-db62-47c4-8d7b-6eb3a3598438"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:56:02.424767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:02.424734    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-xks-ca" pod "ebbb08be-db62-47c4-8d7b-6eb3a3598438" (UID: "ebbb08be-db62-47c4-8d7b-6eb3a3598438"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:56:02.425205 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:02.425183    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbb08be-db62-47c4-8d7b-6eb3a3598438-kube-api-access-cp6bh" pod "ebbb08be-db62-47c4-8d7b-6eb3a3598438" (UID: "ebbb08be-db62-47c4-8d7b-6eb3a3598438"). InnerVolumeSpecName "kube-api-access-cp6bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:56:02.524850 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:02.524807    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:02.524850 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:02.524843    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp6bh\" (UniqueName: \"kubernetes.io/projected/ebbb08be-db62-47c4-8d7b-6eb3a3598438-kube-api-access-cp6bh\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:02.524850 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:02.524854    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebbb08be-db62-47c4-8d7b-6eb3a3598438-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:03.154682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:03.154648    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"
Nov 02 09:56:03.154891 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:03.154693    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4" event={"ID":"ebbb08be-db62-47c4-8d7b-6eb3a3598438","Type":"ContainerDied","Data":"042373a604b8640e14709a28d3a5121f75856e48d8e19da1f344438ad0968207"}
Nov 02 09:56:03.154891 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:03.154760    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="042373a604b8640e14709a28d3a5121f75856e48d8e19da1f344438ad0968207"
Nov 02 09:56:03.242399 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:03.242357    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a15304e-8f78-4fda-a6dc-8f2cd9a22afb" path="/var/lib/kubelet/pods/0a15304e-8f78-4fda-a6dc-8f2cd9a22afb/volumes"
Nov 02 09:56:06.276870 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:06.276826    4449 generic.go:356] "Generic (PLEG): container finished" podID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerID="9c05a6a210cbcd646920b0281a8e82733cbace7b3af3610e1a2f02ab3bff1539" exitCode=0
Nov 02 09:56:06.278013 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:06.277977    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" event={"ID":"fa471864-8cf5-4032-8a6d-c6c61eece6fc","Type":"ContainerDied","Data":"9c05a6a210cbcd646920b0281a8e82733cbace7b3af3610e1a2f02ab3bff1539"}
Nov 02 09:56:06.279969 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:06.279803    4449 scope.go:122] "RemoveContainer" containerID="9c05a6a210cbcd646920b0281a8e82733cbace7b3af3610e1a2f02ab3bff1539"
Nov 02 09:56:06.283415 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:06.283368    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:56:06.284596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:06.284566    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:56:06.284787 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:06.284767    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:56:07.007313 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:56:07.006026    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:56:07.497242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:07.492967    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" event={"ID":"fa471864-8cf5-4032-8a6d-c6c61eece6fc","Type":"ContainerStarted","Data":"622f90c98e95f865d124d4e7b1bb8d67724cd99488c3354e6b4178cca90548d2"}
Nov 02 09:56:08.181243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:08.176472    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:56:08.181243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:08.176654    4449 patch_prober.go:29] interesting pod/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 02 09:56:08.181243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:08.176713    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 02 09:56:09.177244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:09.176688    4449 patch_prober.go:29] interesting pod/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 02 09:56:09.177244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:09.176763    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 02 09:56:09.859247 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:09.841741    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"]
Nov 02 09:56:09.990244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:09.975930    4449 cpu_manager.go:414] "containerMap: removing container" podUID="ebbb08be-db62-47c4-8d7b-6eb3a3598438" containerName="revoke-keys"
Nov 02 09:56:09.990244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:09.975965    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebbb08be-db62-47c4-8d7b-6eb3a3598438" containerName="revoke-keys"
Nov 02 09:56:09.992631 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:09.991256    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebbb08be-db62-47c4-8d7b-6eb3a3598438" containerName="revoke-keys"
Nov 02 09:56:10.129483 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.129390    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"
Nov 02 09:56:10.213365 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.213322    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:56:10.213952 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.213472    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:56:10.213952 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.213551    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:56:10.214303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.214262    4449 patch_prober.go:29] interesting pod/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 02 09:56:10.214545 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.214503    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 02 09:56:10.215055 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.214956    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:56:10.218386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.216137    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wnrb\" (UniqueName: \"kubernetes.io/projected/7169063e-63b8-4f1b-bfba-2d612ab7627b-kube-api-access-8wnrb\") pod \"maas-api-revoke-sub-b4451c93-9tfzm\" (UID: \"7169063e-63b8-4f1b-bfba-2d612ab7627b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"
Nov 02 09:56:10.218386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.216370    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4451c93-9tfzm\" (UID: \"7169063e-63b8-4f1b-bfba-2d612ab7627b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"
Nov 02 09:56:10.218386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.216409    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4451c93-9tfzm\" (UID: \"7169063e-63b8-4f1b-bfba-2d612ab7627b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"
Nov 02 09:56:10.218386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.217288    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"]
Nov 02 09:56:10.353263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.317664    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4451c93-9tfzm\" (UID: \"7169063e-63b8-4f1b-bfba-2d612ab7627b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"
Nov 02 09:56:10.353263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.317728    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4451c93-9tfzm\" (UID: \"7169063e-63b8-4f1b-bfba-2d612ab7627b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"
Nov 02 09:56:10.353263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.318018    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wnrb\" (UniqueName: \"kubernetes.io/projected/7169063e-63b8-4f1b-bfba-2d612ab7627b-kube-api-access-8wnrb\") pod \"maas-api-revoke-sub-b4451c93-9tfzm\" (UID: \"7169063e-63b8-4f1b-bfba-2d612ab7627b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"
Nov 02 09:56:10.353263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.343813    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4451c93-9tfzm\" (UID: \"7169063e-63b8-4f1b-bfba-2d612ab7627b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"
Nov 02 09:56:10.359207 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.359174    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wnrb\" (UniqueName: \"kubernetes.io/projected/7169063e-63b8-4f1b-bfba-2d612ab7627b-kube-api-access-8wnrb\") pod \"maas-api-revoke-sub-b4451c93-9tfzm\" (UID: \"7169063e-63b8-4f1b-bfba-2d612ab7627b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"
Nov 02 09:56:10.365245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.361767    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4451c93-9tfzm\" (UID: \"7169063e-63b8-4f1b-bfba-2d612ab7627b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"
Nov 02 09:56:10.381298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.378609    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"
Nov 02 09:56:10.636313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.631329    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:56:10.878241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:10.874379    4449 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:11.019757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:11.019330    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:11.206375 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:11.206335    4449 patch_prober.go:29] interesting pod/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 02 09:56:11.206638 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:11.206608    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 02 09:56:11.336285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:11.335608    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:11.336285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:11.336023    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 09:56:11.348252 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:11.337086    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n" podUID="7968e57e-6e72-4ef2-af9f-82c111d870f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.162:8000/ready\": context deadline exceeded"
Nov 02 09:56:11.525242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:11.522946    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:12.028310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.023399    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:12.028310 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.024635    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:56:12.127243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.101630    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"]
Nov 02 09:56:12.235778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.235735    4449 patch_prober.go:29] interesting pod/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 02 09:56:12.236097 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.236049    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 02 09:56:12.247331 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.247164    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:56:12.342012 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.341969    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w" podUID="6e5b3557-3492-4152-9e0a-50291c7d1853" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:12.343206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.343184    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:56:12.404241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.402156    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.156:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:12.404241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.402955    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:56:12.495435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.493506    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w"
Nov 02 09:56:12.507664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.507630    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm" event={"ID":"7169063e-63b8-4f1b-bfba-2d612ab7627b","Type":"ContainerStarted","Data":"f350924002af721339c9afc7bd33211fe7bdd0215c1b168b185067ca659feaa1"}
Nov 02 09:56:12.512714 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.512688    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"]
Nov 02 09:56:12.528240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.521139    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:56:12.528240 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:12.523491    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9308a5-hs7ss"]
Nov 02 09:56:13.174654 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:13.174614    4449 patch_prober.go:29] interesting pod/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 02 09:56:13.174964 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:13.174932    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 02 09:56:13.208502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:13.208446    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf1721c3-3b9c-49c3-b032-b2a46efe518a" path="/var/lib/kubelet/pods/bf1721c3-3b9c-49c3-b032-b2a46efe518a/volumes"
Nov 02 09:56:13.518594 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:13.518497    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm" event={"ID":"7169063e-63b8-4f1b-bfba-2d612ab7627b","Type":"ContainerStarted","Data":"1977f75ce70c98001bf1d50fb2e160bf8ff3157cbb9ea29cfd1440ba6185901a"}
Nov 02 09:56:13.538686 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:13.538640    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm" podStartSLOduration=4.538626391 podStartE2EDuration="4.538626391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:56:13.535806124 +0000 UTC m=+1250.474092901" watchObservedRunningTime="2026-11-02 09:56:13.538626391 +0000 UTC m=+1250.476913165"
Nov 02 09:56:14.171486 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:14.171451    4449 patch_prober.go:29] interesting pod/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Nov 02 09:56:14.171691 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:14.171503    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Nov 02 09:56:14.541046 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:14.541009    4449 generic.go:356] "Generic (PLEG): container finished" podID="7169063e-63b8-4f1b-bfba-2d612ab7627b" containerID="1977f75ce70c98001bf1d50fb2e160bf8ff3157cbb9ea29cfd1440ba6185901a" exitCode=0
Nov 02 09:56:14.541528 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:14.541085    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm" event={"ID":"7169063e-63b8-4f1b-bfba-2d612ab7627b","Type":"ContainerDied","Data":"1977f75ce70c98001bf1d50fb2e160bf8ff3157cbb9ea29cfd1440ba6185901a"}
Nov 02 09:56:15.176201 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:15.176169    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:56:15.177327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:15.177302    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:56:15.723592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:15.723564    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"
Nov 02 09:56:15.776859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:15.776832    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7169063e-63b8-4f1b-bfba-2d612ab7627b-kube-api-access-8wnrb\" (UniqueName: \"kubernetes.io/projected/7169063e-63b8-4f1b-bfba-2d612ab7627b-kube-api-access-8wnrb\") pod \"7169063e-63b8-4f1b-bfba-2d612ab7627b\" (UID: \"7169063e-63b8-4f1b-bfba-2d612ab7627b\") "
Nov 02 09:56:15.777166 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:15.777145    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-xks-ca\") pod \"7169063e-63b8-4f1b-bfba-2d612ab7627b\" (UID: \"7169063e-63b8-4f1b-bfba-2d612ab7627b\") "
Nov 02 09:56:15.777277 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:15.777191    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-openshift-ca\") pod \"7169063e-63b8-4f1b-bfba-2d612ab7627b\" (UID: \"7169063e-63b8-4f1b-bfba-2d612ab7627b\") "
Nov 02 09:56:15.777695 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:15.777663    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-openshift-ca" pod "7169063e-63b8-4f1b-bfba-2d612ab7627b" (UID: "7169063e-63b8-4f1b-bfba-2d612ab7627b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:56:15.779444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:15.779420    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-xks-ca" pod "7169063e-63b8-4f1b-bfba-2d612ab7627b" (UID: "7169063e-63b8-4f1b-bfba-2d612ab7627b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:56:15.779687 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:15.779660    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7169063e-63b8-4f1b-bfba-2d612ab7627b-kube-api-access-8wnrb" pod "7169063e-63b8-4f1b-bfba-2d612ab7627b" (UID: "7169063e-63b8-4f1b-bfba-2d612ab7627b"). InnerVolumeSpecName "kube-api-access-8wnrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:56:15.878874 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:15.878774    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:15.878874 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:15.878811    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7169063e-63b8-4f1b-bfba-2d612ab7627b-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:15.878874 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:15.878826    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wnrb\" (UniqueName: \"kubernetes.io/projected/7169063e-63b8-4f1b-bfba-2d612ab7627b-kube-api-access-8wnrb\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:16.562324 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:16.562278    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"
Nov 02 09:56:16.562503 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:16.562338    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm" event={"ID":"7169063e-63b8-4f1b-bfba-2d612ab7627b","Type":"ContainerDied","Data":"f350924002af721339c9afc7bd33211fe7bdd0215c1b168b185067ca659feaa1"}
Nov 02 09:56:16.562503 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:16.562383    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f350924002af721339c9afc7bd33211fe7bdd0215c1b168b185067ca659feaa1"
Nov 02 09:56:20.212550 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.212513    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"]
Nov 02 09:56:20.213235 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.213203    4449 cpu_manager.go:414] "containerMap: removing container" podUID="7169063e-63b8-4f1b-bfba-2d612ab7627b" containerName="revoke-keys"
Nov 02 09:56:20.213235 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.213233    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7169063e-63b8-4f1b-bfba-2d612ab7627b" containerName="revoke-keys"
Nov 02 09:56:20.213452 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.213440    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="7169063e-63b8-4f1b-bfba-2d612ab7627b" containerName="revoke-keys"
Nov 02 09:56:20.217332 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.217312    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"
Nov 02 09:56:20.221932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.221824    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:56:20.222724 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.222702    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:56:20.222724 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.222724    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:56:20.223029 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.223007    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:56:20.226457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.226430    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"]
Nov 02 09:56:20.380529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.380495    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac98a2-nfjx8\" (UID: \"b7251496-4b91-4330-9eb1-f6c91492f2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"
Nov 02 09:56:20.380529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.380531    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac98a2-nfjx8\" (UID: \"b7251496-4b91-4330-9eb1-f6c91492f2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"
Nov 02 09:56:20.380752 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.380659    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhfd9\" (UniqueName: \"kubernetes.io/projected/b7251496-4b91-4330-9eb1-f6c91492f2aa-kube-api-access-xhfd9\") pod \"maas-api-revoke-sub-5bac98a2-nfjx8\" (UID: \"b7251496-4b91-4330-9eb1-f6c91492f2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"
Nov 02 09:56:20.482298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.482193    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac98a2-nfjx8\" (UID: \"b7251496-4b91-4330-9eb1-f6c91492f2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"
Nov 02 09:56:20.482298 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.482273    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac98a2-nfjx8\" (UID: \"b7251496-4b91-4330-9eb1-f6c91492f2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"
Nov 02 09:56:20.482492 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.482366    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhfd9\" (UniqueName: \"kubernetes.io/projected/b7251496-4b91-4330-9eb1-f6c91492f2aa-kube-api-access-xhfd9\") pod \"maas-api-revoke-sub-5bac98a2-nfjx8\" (UID: \"b7251496-4b91-4330-9eb1-f6c91492f2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"
Nov 02 09:56:20.482938 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.482914    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac98a2-nfjx8\" (UID: \"b7251496-4b91-4330-9eb1-f6c91492f2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"
Nov 02 09:56:20.484902 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.484875    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac98a2-nfjx8\" (UID: \"b7251496-4b91-4330-9eb1-f6c91492f2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"
Nov 02 09:56:20.494711 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.494678    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhfd9\" (UniqueName: \"kubernetes.io/projected/b7251496-4b91-4330-9eb1-f6c91492f2aa-kube-api-access-xhfd9\") pod \"maas-api-revoke-sub-5bac98a2-nfjx8\" (UID: \"b7251496-4b91-4330-9eb1-f6c91492f2aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"
Nov 02 09:56:20.534371 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.534339    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"
Nov 02 09:56:20.826747 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:20.826612    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"]
Nov 02 09:56:21.692873 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:21.691617    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8" event={"ID":"b7251496-4b91-4330-9eb1-f6c91492f2aa","Type":"ContainerStarted","Data":"49cb4eeeb605f47ba8a63fa42196d552592381b596a740b6d58c065c78752ad2"}
Nov 02 09:56:21.692873 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:21.691698    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8" event={"ID":"b7251496-4b91-4330-9eb1-f6c91492f2aa","Type":"ContainerStarted","Data":"2dae477143c2e5f6cb9c7f1885737fe0235e91e75dcd5bfc09a636d20e49ccca"}
Nov 02 09:56:21.757696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:21.757631    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8" podStartSLOduration=1.7576123639999999 podStartE2EDuration="1.757612364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:56:21.73448787 +0000 UTC m=+1258.672774648" watchObservedRunningTime="2026-11-02 09:56:21.757612364 +0000 UTC m=+1258.695899141"
Nov 02 09:56:22.702435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:22.702404    4449 generic.go:356] "Generic (PLEG): container finished" podID="b7251496-4b91-4330-9eb1-f6c91492f2aa" containerID="49cb4eeeb605f47ba8a63fa42196d552592381b596a740b6d58c065c78752ad2" exitCode=0
Nov 02 09:56:22.702844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:22.702484    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8" event={"ID":"b7251496-4b91-4330-9eb1-f6c91492f2aa","Type":"ContainerDied","Data":"49cb4eeeb605f47ba8a63fa42196d552592381b596a740b6d58c065c78752ad2"}
Nov 02 09:56:23.886226 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:23.886191    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"
Nov 02 09:56:24.029751 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.029659    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"]
Nov 02 09:56:24.036302 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.036273    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f9de60-s94gj"]
Nov 02 09:56:24.047313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.047284    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-openshift-ca\") pod \"b7251496-4b91-4330-9eb1-f6c91492f2aa\" (UID: \"b7251496-4b91-4330-9eb1-f6c91492f2aa\") "
Nov 02 09:56:24.047489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.047423    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-xks-ca\") pod \"b7251496-4b91-4330-9eb1-f6c91492f2aa\" (UID: \"b7251496-4b91-4330-9eb1-f6c91492f2aa\") "
Nov 02 09:56:24.047556 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.047525    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7251496-4b91-4330-9eb1-f6c91492f2aa-kube-api-access-xhfd9\" (UniqueName: \"kubernetes.io/projected/b7251496-4b91-4330-9eb1-f6c91492f2aa-kube-api-access-xhfd9\") pod \"b7251496-4b91-4330-9eb1-f6c91492f2aa\" (UID: \"b7251496-4b91-4330-9eb1-f6c91492f2aa\") "
Nov 02 09:56:24.047715 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.047690    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-openshift-ca" pod "b7251496-4b91-4330-9eb1-f6c91492f2aa" (UID: "b7251496-4b91-4330-9eb1-f6c91492f2aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:56:24.048167 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.048090    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:24.049934 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.049905    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7251496-4b91-4330-9eb1-f6c91492f2aa-kube-api-access-xhfd9" pod "b7251496-4b91-4330-9eb1-f6c91492f2aa" (UID: "b7251496-4b91-4330-9eb1-f6c91492f2aa"). InnerVolumeSpecName "kube-api-access-xhfd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:56:24.050022 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.049974    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-xks-ca" pod "b7251496-4b91-4330-9eb1-f6c91492f2aa" (UID: "b7251496-4b91-4330-9eb1-f6c91492f2aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:56:24.149338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.149286    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7251496-4b91-4330-9eb1-f6c91492f2aa-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:24.149338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.149342    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhfd9\" (UniqueName: \"kubernetes.io/projected/b7251496-4b91-4330-9eb1-f6c91492f2aa-kube-api-access-xhfd9\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:24.458939 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.458911    4449 scope.go:122] "RemoveContainer" containerID="ab9ed78d7ac61f39e633a17f875f84269710412052a0863fee7f6ba1338bf0e4"
Nov 02 09:56:24.473554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.473485    4449 scope.go:122] "RemoveContainer" containerID="2e9732213da9391c2688bf06e273874581154966e46ae1c2c4fda70911661b90"
Nov 02 09:56:24.485547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.485520    4449 scope.go:122] "RemoveContainer" containerID="7b95233ee9918fdb7ef4ad32ed0f79280b5ce4fd9fe95b6a24201839f54a4247"
Nov 02 09:56:24.499770 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.499747    4449 scope.go:122] "RemoveContainer" containerID="eb98dcb55a70e698a2c3d3aa47d43ca7b5de9eddf5916b1daf214ad9b161e725"
Nov 02 09:56:24.510645 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.510623    4449 scope.go:122] "RemoveContainer" containerID="f896790bee3f6ccc535c4b12fa42fbce219b8bf1619422fd7ded5ac7b6ada115"
Nov 02 09:56:24.520671 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.520650    4449 scope.go:122] "RemoveContainer" containerID="932bdf6028e7fbdbe07b7c4ee3c4f97dabbc85bfa2fadc3956d2dd16b2c18dce"
Nov 02 09:56:24.530650 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.530630    4449 scope.go:122] "RemoveContainer" containerID="6b1a78e10b68330aa65d7d0aad18f38c185be0ff63452c9e4b23cc8978300e63"
Nov 02 09:56:24.720830 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.720753    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"
Nov 02 09:56:24.720830 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.720776    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8" event={"ID":"b7251496-4b91-4330-9eb1-f6c91492f2aa","Type":"ContainerDied","Data":"2dae477143c2e5f6cb9c7f1885737fe0235e91e75dcd5bfc09a636d20e49ccca"}
Nov 02 09:56:24.721020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:24.720855    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dae477143c2e5f6cb9c7f1885737fe0235e91e75dcd5bfc09a636d20e49ccca"
Nov 02 09:56:25.207473 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:25.207434    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8be54d-ca5b-4794-b516-7b14c36bc840" path="/var/lib/kubelet/pods/0e8be54d-ca5b-4794-b516-7b14c36bc840/volumes"
Nov 02 09:56:31.089252 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:31.050048    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"]
Nov 02 09:56:32.926349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:32.926300    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"]
Nov 02 09:56:34.027718 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:34.027681    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"]
Nov 02 09:56:34.032458 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:34.032425    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4893be-42r6d"]
Nov 02 09:56:35.209124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:35.209087    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7fefd8-6d22-464a-890d-40983e8e9288" path="/var/lib/kubelet/pods/7a7fefd8-6d22-464a-890d-40983e8e9288/volumes"
Nov 02 09:56:36.547673 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:56:36.547640    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:56:37.031149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:37.031052    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"]
Nov 02 09:56:37.035858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:37.035828    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6d6713-9cnnl"]
Nov 02 09:56:37.208534 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:37.208501    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4674912-6f57-42d8-b948-a36fd412d1b7" path="/var/lib/kubelet/pods/c4674912-6f57-42d8-b948-a36fd412d1b7/volumes"
Nov 02 09:56:40.028993 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:40.028954    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"]
Nov 02 09:56:40.033899 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:40.033873    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b9baa7e-qvmrc"]
Nov 02 09:56:41.208922 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:41.208884    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a524abe-238d-4fcf-a096-d07032f33da4" path="/var/lib/kubelet/pods/6a524abe-238d-4fcf-a096-d07032f33da4/volumes"
Nov 02 09:56:46.425276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.421250    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerName="main" containerID="cri-o://13f9ec5a3c50fd979f836b85b296544e5a668ca60150b4ed711cf5e3e0fed402" gracePeriod=45
Nov 02 09:56:46.522467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.522418    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.156:8000/ready\": dial tcp 10.134.0.156:8000: connect: connection refused"
Nov 02 09:56:46.717795 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.717767    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:56:46.819388 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.818879    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tls-certs\") pod \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") "
Nov 02 09:56:46.819690 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.819648    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-dshm\") pod \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") "
Nov 02 09:56:46.820613 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.820284    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-model-cache\") pod \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") "
Nov 02 09:56:46.820861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.820567    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-model-cache" pod "f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" (UID: "f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:46.821013 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.820996    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-kube-api-access-rqwkf\" (UniqueName: \"kubernetes.io/projected/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-kube-api-access-rqwkf\") pod \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") "
Nov 02 09:56:46.821980 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.821938    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tmp-dir" pod "f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" (UID: "f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:46.822096 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.821688    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tmp-dir\") pod \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") "
Nov 02 09:56:46.822527 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.822287    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-home\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-home\") pod \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\" (UID: \"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9\") "
Nov 02 09:56:46.822771 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.822512    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-home" pod "f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" (UID: "f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:46.823519 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.823483    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:46.823642 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.823631    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:46.823726 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.823716    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:46.843844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.843115    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-kube-api-access-rqwkf" pod "f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" (UID: "f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9"). InnerVolumeSpecName "kube-api-access-rqwkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:56:46.877311 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.871718    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-dshm" pod "f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" (UID: "f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:46.877311 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.872315    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tls-certs" pod "f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" (UID: "f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:56:46.926732 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.926674    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:46.926732 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.926718    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:46.926732 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.926736    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqwkf\" (UniqueName: \"kubernetes.io/projected/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9-kube-api-access-rqwkf\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:46.970206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.965363    4449 generic.go:356] "Generic (PLEG): container finished" podID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerID="13f9ec5a3c50fd979f836b85b296544e5a668ca60150b4ed711cf5e3e0fed402" exitCode=0
Nov 02 09:56:46.970206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.965490    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" event={"ID":"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9","Type":"ContainerDied","Data":"13f9ec5a3c50fd979f836b85b296544e5a668ca60150b4ed711cf5e3e0fed402"}
Nov 02 09:56:46.970206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.965530    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d" event={"ID":"f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9","Type":"ContainerDied","Data":"aa607e8fc017f3afa53f3cecf10dc67d5ba357c9d6975326b56befca2891f556"}
Nov 02 09:56:46.970206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.965553    4449 scope.go:122] "RemoveContainer" containerID="13f9ec5a3c50fd979f836b85b296544e5a668ca60150b4ed711cf5e3e0fed402"
Nov 02 09:56:46.970206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:46.965704    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"
Nov 02 09:56:47.023371 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:47.014513    4449 scope.go:122] "RemoveContainer" containerID="13f9ec5a3c50fd979f836b85b296544e5a668ca60150b4ed711cf5e3e0fed402"
Nov 02 09:56:47.040740 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:56:47.025394    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f9ec5a3c50fd979f836b85b296544e5a668ca60150b4ed711cf5e3e0fed402\": container with ID starting with 13f9ec5a3c50fd979f836b85b296544e5a668ca60150b4ed711cf5e3e0fed402 not found: ID does not exist" containerID="13f9ec5a3c50fd979f836b85b296544e5a668ca60150b4ed711cf5e3e0fed402"
Nov 02 09:56:47.040740 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:47.025445    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f9ec5a3c50fd979f836b85b296544e5a668ca60150b4ed711cf5e3e0fed402"} err="failed to get container status \"13f9ec5a3c50fd979f836b85b296544e5a668ca60150b4ed711cf5e3e0fed402\": rpc error: code = NotFound desc = could not find container \"13f9ec5a3c50fd979f836b85b296544e5a668ca60150b4ed711cf5e3e0fed402\": container with ID starting with 13f9ec5a3c50fd979f836b85b296544e5a668ca60150b4ed711cf5e3e0fed402 not found: ID does not exist"
Nov 02 09:56:47.070886 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:47.070847    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"]
Nov 02 09:56:47.084735 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:47.084701    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/sub-test-model-bde944-kserve-55d5f8c9b5-w6d5d"]
Nov 02 09:56:47.211941 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:47.211831    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" path="/var/lib/kubelet/pods/f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9/volumes"
Nov 02 09:56:47.950064 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:47.950026    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n" podUID="7968e57e-6e72-4ef2-af9f-82c111d870f8" containerName="main" containerID="cri-o://7526d32fbc216ad1d9888229b3a6895310c12504ef4b97814935fdd8b8125c77" gracePeriod=45
Nov 02 09:56:48.311969 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.311933    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:56:48.421328 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.420941    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-model-cache\") pod \"7968e57e-6e72-4ef2-af9f-82c111d870f8\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") "
Nov 02 09:56:48.421328 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.421026    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-home\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-home\") pod \"7968e57e-6e72-4ef2-af9f-82c111d870f8\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") "
Nov 02 09:56:48.421328 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.421101    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7968e57e-6e72-4ef2-af9f-82c111d870f8-kube-api-access-6gh7v\" (UniqueName: \"kubernetes.io/projected/7968e57e-6e72-4ef2-af9f-82c111d870f8-kube-api-access-6gh7v\") pod \"7968e57e-6e72-4ef2-af9f-82c111d870f8\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") "
Nov 02 09:56:48.421328 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.421148    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-tmp-dir\") pod \"7968e57e-6e72-4ef2-af9f-82c111d870f8\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") "
Nov 02 09:56:48.421328 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.421232    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-dshm\") pod \"7968e57e-6e72-4ef2-af9f-82c111d870f8\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") "
Nov 02 09:56:48.421976 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.421942    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-home" pod "7968e57e-6e72-4ef2-af9f-82c111d870f8" (UID: "7968e57e-6e72-4ef2-af9f-82c111d870f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:48.422180 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.422154    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-model-cache" pod "7968e57e-6e72-4ef2-af9f-82c111d870f8" (UID: "7968e57e-6e72-4ef2-af9f-82c111d870f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:48.422389 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.422365    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7968e57e-6e72-4ef2-af9f-82c111d870f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/7968e57e-6e72-4ef2-af9f-82c111d870f8-tls-certs\") pod \"7968e57e-6e72-4ef2-af9f-82c111d870f8\" (UID: \"7968e57e-6e72-4ef2-af9f-82c111d870f8\") "
Nov 02 09:56:48.430254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.422587    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-tmp-dir" pod "7968e57e-6e72-4ef2-af9f-82c111d870f8" (UID: "7968e57e-6e72-4ef2-af9f-82c111d870f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:48.430254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.423396    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:48.430254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.423422    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:48.430254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.423440    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:48.430254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.425128    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-dshm" pod "7968e57e-6e72-4ef2-af9f-82c111d870f8" (UID: "7968e57e-6e72-4ef2-af9f-82c111d870f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:48.430254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.428273    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7968e57e-6e72-4ef2-af9f-82c111d870f8-kube-api-access-6gh7v" pod "7968e57e-6e72-4ef2-af9f-82c111d870f8" (UID: "7968e57e-6e72-4ef2-af9f-82c111d870f8"). InnerVolumeSpecName "kube-api-access-6gh7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:56:48.434466 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.434204    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7968e57e-6e72-4ef2-af9f-82c111d870f8-tls-certs" pod "7968e57e-6e72-4ef2-af9f-82c111d870f8" (UID: "7968e57e-6e72-4ef2-af9f-82c111d870f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:56:48.525350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.525245    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gh7v\" (UniqueName: \"kubernetes.io/projected/7968e57e-6e72-4ef2-af9f-82c111d870f8-kube-api-access-6gh7v\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:48.525350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.525284    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7968e57e-6e72-4ef2-af9f-82c111d870f8-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:48.525350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:48.525298    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7968e57e-6e72-4ef2-af9f-82c111d870f8-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:49.002948 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:49.002910    4449 generic.go:356] "Generic (PLEG): container finished" podID="7968e57e-6e72-4ef2-af9f-82c111d870f8" containerID="7526d32fbc216ad1d9888229b3a6895310c12504ef4b97814935fdd8b8125c77" exitCode=0
Nov 02 09:56:49.003665 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:49.003018    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n" event={"ID":"7968e57e-6e72-4ef2-af9f-82c111d870f8","Type":"ContainerDied","Data":"7526d32fbc216ad1d9888229b3a6895310c12504ef4b97814935fdd8b8125c77"}
Nov 02 09:56:49.003665 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:49.003070    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n" event={"ID":"7968e57e-6e72-4ef2-af9f-82c111d870f8","Type":"ContainerDied","Data":"6e96678041dde005fc27e2e8f01891aae8916db076dcd4b251c5b3388549befb"}
Nov 02 09:56:49.003665 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:49.003094    4449 scope.go:122] "RemoveContainer" containerID="7526d32fbc216ad1d9888229b3a6895310c12504ef4b97814935fdd8b8125c77"
Nov 02 09:56:49.003665 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:49.003306    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"
Nov 02 09:56:49.124339 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:49.072877    4449 scope.go:122] "RemoveContainer" containerID="7526d32fbc216ad1d9888229b3a6895310c12504ef4b97814935fdd8b8125c77"
Nov 02 09:56:49.124339 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:49.077429    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"]
Nov 02 09:56:49.124339 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:56:49.078619    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7526d32fbc216ad1d9888229b3a6895310c12504ef4b97814935fdd8b8125c77\": container with ID starting with 7526d32fbc216ad1d9888229b3a6895310c12504ef4b97814935fdd8b8125c77 not found: ID does not exist" containerID="7526d32fbc216ad1d9888229b3a6895310c12504ef4b97814935fdd8b8125c77"
Nov 02 09:56:49.124339 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:49.078666    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7526d32fbc216ad1d9888229b3a6895310c12504ef4b97814935fdd8b8125c77"} err="failed to get container status \"7526d32fbc216ad1d9888229b3a6895310c12504ef4b97814935fdd8b8125c77\": rpc error: code = NotFound desc = could not find container \"7526d32fbc216ad1d9888229b3a6895310c12504ef4b97814935fdd8b8125c77\": container with ID starting with 7526d32fbc216ad1d9888229b3a6895310c12504ef4b97814935fdd8b8125c77 not found: ID does not exist"
Nov 02 09:56:49.124339 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:49.112674    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/sub-test-model-9e7156-kserve-74bb6878f5-rn88n"]
Nov 02 09:56:49.281296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:49.276096    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7968e57e-6e72-4ef2-af9f-82c111d870f8" path="/var/lib/kubelet/pods/7968e57e-6e72-4ef2-af9f-82c111d870f8/volumes"
Nov 02 09:56:50.034228 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:50.034172    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"]
Nov 02 09:56:50.039042 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:50.039015    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d1d900-2vhcp"]
Nov 02 09:56:51.234245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:51.234141    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc77c739-002f-4c2a-bdc4-e68ef8472812" path="/var/lib/kubelet/pods/dc77c739-002f-4c2a-bdc4-e68ef8472812/volumes"
Nov 02 09:56:53.356853 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.356814    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"]
Nov 02 09:56:53.357592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.357573    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerName="main"
Nov 02 09:56:53.357592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.357593    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerName="main"
Nov 02 09:56:53.357684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.357606    4449 cpu_manager.go:414] "containerMap: removing container" podUID="7968e57e-6e72-4ef2-af9f-82c111d870f8" containerName="main"
Nov 02 09:56:53.357684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.357611    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7968e57e-6e72-4ef2-af9f-82c111d870f8" containerName="main"
Nov 02 09:56:53.357684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.357629    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b7251496-4b91-4330-9eb1-f6c91492f2aa" containerName="revoke-keys"
Nov 02 09:56:53.357684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.357635    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7251496-4b91-4330-9eb1-f6c91492f2aa" containerName="revoke-keys"
Nov 02 09:56:53.357821 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.357757    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7251496-4b91-4330-9eb1-f6c91492f2aa" containerName="revoke-keys"
Nov 02 09:56:53.357821 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.357773    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cec0e5-a1f2-4ebe-b72b-5ef446cfdeb9" containerName="main"
Nov 02 09:56:53.357821 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.357780    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="7968e57e-6e72-4ef2-af9f-82c111d870f8" containerName="main"
Nov 02 09:56:53.363237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.363199    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:53.371610 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.371578    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"]
Nov 02 09:56:53.371794 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.371737    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 09:56:53.371794 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.371745    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 09:56:53.371901 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.371787    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-993442311390\""
Nov 02 09:56:53.372770 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.372752    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 09:56:53.373069 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.373052    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 09:56:53.373121 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.373063    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-xdvq7\""
Nov 02 09:56:53.527598 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.520768    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q45qn\" (UniqueName: \"kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-kube-api-access-q45qn\") pod \"payload-processing-external-model-87c8cf6-sftdh\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") " pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:53.527598 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.520825    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-routing-overlay-volume\") pod \"payload-processing-external-model-87c8cf6-sftdh\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") " pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:53.527598 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.520858    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-plugins-config-volume\") pod \"payload-processing-external-model-87c8cf6-sftdh\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") " pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:53.527598 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.520890    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-provider-credentials\") pod \"payload-processing-external-model-87c8cf6-sftdh\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") " pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:53.622715 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.622627    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45qn\" (UniqueName: \"kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-kube-api-access-q45qn\") pod \"payload-processing-external-model-87c8cf6-sftdh\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") " pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:53.622715 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.622667    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-routing-overlay-volume\") pod \"payload-processing-external-model-87c8cf6-sftdh\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") " pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:53.622715 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.622698    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-plugins-config-volume\") pod \"payload-processing-external-model-87c8cf6-sftdh\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") " pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:53.623011 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.622731    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-provider-credentials\") pod \"payload-processing-external-model-87c8cf6-sftdh\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") " pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:53.623383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.623360    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-routing-overlay-volume\") pod \"payload-processing-external-model-87c8cf6-sftdh\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") " pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:53.623557 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.623535    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-plugins-config-volume\") pod \"payload-processing-external-model-87c8cf6-sftdh\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") " pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:53.625385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.625364    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-provider-credentials\") pod \"payload-processing-external-model-87c8cf6-sftdh\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") " pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:53.636243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.636193    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q45qn\" (UniqueName: \"kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-kube-api-access-q45qn\") pod \"payload-processing-external-model-87c8cf6-sftdh\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") " pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:53.688248 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:53.681503    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:54.082053 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:56:54.082005    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085247d8_11ce_41b8_9ef7_0f940cab26fd.slice/crio-a3168da54600c0e29521bdaac98b967318a17695587db61c9d156db2a1203aa4 WatchSource:0}: Error finding container a3168da54600c0e29521bdaac98b967318a17695587db61c9d156db2a1203aa4: Status 404 returned error can't find the container with id a3168da54600c0e29521bdaac98b967318a17695587db61c9d156db2a1203aa4
Nov 02 09:56:54.083336 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:54.083306    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"]
Nov 02 09:56:54.149818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:54.149781    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh" event={"ID":"085247d8-11ce-41b8-9ef7-0f940cab26fd","Type":"ContainerStarted","Data":"a3168da54600c0e29521bdaac98b967318a17695587db61c9d156db2a1203aa4"}
Nov 02 09:56:55.168074 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:55.168035    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh" event={"ID":"085247d8-11ce-41b8-9ef7-0f940cab26fd","Type":"ContainerStarted","Data":"cbfb3455f6288a17e65600569ebbf59ce53cddad806d27f73a56e5003f8fec8e"}
Nov 02 09:56:55.168596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:55.168290    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:56:55.190580 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:56:55.190526    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh" podStartSLOduration=2.190511717 podStartE2EDuration="2.190511717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:56:55.186349143 +0000 UTC m=+1292.124635938" watchObservedRunningTime="2026-11-02 09:56:55.190511717 +0000 UTC m=+1292.128798493"
Nov 02 09:57:01.936907 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:01.936855    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"]
Nov 02 09:57:01.938440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:01.938100    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy" containerID="cri-o://622f90c98e95f865d124d4e7b1bb8d67724cd99488c3354e6b4178cca90548d2" gracePeriod=30
Nov 02 09:57:05.038034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:05.037946    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"]
Nov 02 09:57:05.043142 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:05.043109    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63067f92-nrvtz"]
Nov 02 09:57:05.233287 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:05.233240    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29a1545-bff1-42a5-9332-df3922620bb1" path="/var/lib/kubelet/pods/a29a1545-bff1-42a5-9332-df3922620bb1/volumes"
Nov 02 09:57:06.179887 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:06.179850    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:57:06.548299 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:57:06.548262    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:57:07.265987 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.265958    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:57:07.328647 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.328608    4449 generic.go:356] "Generic (PLEG): container finished" podID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerID="622f90c98e95f865d124d4e7b1bb8d67724cd99488c3354e6b4178cca90548d2" exitCode=0
Nov 02 09:57:07.328871 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.328844    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" event={"ID":"fa471864-8cf5-4032-8a6d-c6c61eece6fc","Type":"ContainerDied","Data":"622f90c98e95f865d124d4e7b1bb8d67724cd99488c3354e6b4178cca90548d2"}
Nov 02 09:57:07.328953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.328914    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv" event={"ID":"fa471864-8cf5-4032-8a6d-c6c61eece6fc","Type":"ContainerDied","Data":"404077af4a2888bf3f29b32a0f8c0cf5564524e3abdd275356b2aabae1d242d7"}
Nov 02 09:57:07.328953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.328948    4449 scope.go:122] "RemoveContainer" containerID="622f90c98e95f865d124d4e7b1bb8d67724cd99488c3354e6b4178cca90548d2"
Nov 02 09:57:07.329174 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.329155    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"
Nov 02 09:57:07.352909 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.352882    4449 scope.go:122] "RemoveContainer" containerID="9c05a6a210cbcd646920b0281a8e82733cbace7b3af3610e1a2f02ab3bff1539"
Nov 02 09:57:07.362105 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.362073    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-podinfo\") pod \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") "
Nov 02 09:57:07.362701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.362137    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istiod-ca-cert\") pod \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") "
Nov 02 09:57:07.362701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.362180    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-envoy\") pod \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") "
Nov 02 09:57:07.362701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.362264    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-kube-api-access-96v6s\" (UniqueName: \"kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-kube-api-access-96v6s\") pod \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") "
Nov 02 09:57:07.362701 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.362685    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-credential-socket" pod "fa471864-8cf5-4032-8a6d-c6c61eece6fc" (UID: "fa471864-8cf5-4032-8a6d-c6c61eece6fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:07.363012 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.362982    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istiod-ca-cert" pod "fa471864-8cf5-4032-8a6d-c6c61eece6fc" (UID: "fa471864-8cf5-4032-8a6d-c6c61eece6fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:57:07.363099 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.363061    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-credential-socket\") pod \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") "
Nov 02 09:57:07.363161 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.363129    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-certs\") pod \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") "
Nov 02 09:57:07.363371 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.363352    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-certs" pod "fa471864-8cf5-4032-8a6d-c6c61eece6fc" (UID: "fa471864-8cf5-4032-8a6d-c6c61eece6fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:07.363528 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.363511    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-data\") pod \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") "
Nov 02 09:57:07.363692 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.363668    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-token\" (UniqueName: \"kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-token\") pod \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") "
Nov 02 09:57:07.364621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.364596    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-data" pod "fa471864-8cf5-4032-8a6d-c6c61eece6fc" (UID: "fa471864-8cf5-4032-8a6d-c6c61eece6fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:07.364807 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.364790    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-socket\") pod \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\" (UID: \"fa471864-8cf5-4032-8a6d-c6c61eece6fc\") "
Nov 02 09:57:07.365167 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.365148    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-socket" pod "fa471864-8cf5-4032-8a6d-c6c61eece6fc" (UID: "fa471864-8cf5-4032-8a6d-c6c61eece6fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:07.365910 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.365789    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:07.366034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.365915    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:07.366034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.365931    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:07.366034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.365944    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:07.366034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.365958    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:07.368650 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.368620    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-envoy" pod "fa471864-8cf5-4032-8a6d-c6c61eece6fc" (UID: "fa471864-8cf5-4032-8a6d-c6c61eece6fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:07.368781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.368750    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-token" pod "fa471864-8cf5-4032-8a6d-c6c61eece6fc" (UID: "fa471864-8cf5-4032-8a6d-c6c61eece6fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:57:07.370339 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.370299    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-kube-api-access-96v6s" pod "fa471864-8cf5-4032-8a6d-c6c61eece6fc" (UID: "fa471864-8cf5-4032-8a6d-c6c61eece6fc"). InnerVolumeSpecName "kube-api-access-96v6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:57:07.370637 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.370613    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-podinfo" pod "fa471864-8cf5-4032-8a6d-c6c61eece6fc" (UID: "fa471864-8cf5-4032-8a6d-c6c61eece6fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:57:07.371505 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.371483    4449 scope.go:122] "RemoveContainer" containerID="622f90c98e95f865d124d4e7b1bb8d67724cd99488c3354e6b4178cca90548d2"
Nov 02 09:57:07.371932 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:57:07.371908    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622f90c98e95f865d124d4e7b1bb8d67724cd99488c3354e6b4178cca90548d2\": container with ID starting with 622f90c98e95f865d124d4e7b1bb8d67724cd99488c3354e6b4178cca90548d2 not found: ID does not exist" containerID="622f90c98e95f865d124d4e7b1bb8d67724cd99488c3354e6b4178cca90548d2"
Nov 02 09:57:07.372024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.371943    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622f90c98e95f865d124d4e7b1bb8d67724cd99488c3354e6b4178cca90548d2"} err="failed to get container status \"622f90c98e95f865d124d4e7b1bb8d67724cd99488c3354e6b4178cca90548d2\": rpc error: code = NotFound desc = could not find container \"622f90c98e95f865d124d4e7b1bb8d67724cd99488c3354e6b4178cca90548d2\": container with ID starting with 622f90c98e95f865d124d4e7b1bb8d67724cd99488c3354e6b4178cca90548d2 not found: ID does not exist"
Nov 02 09:57:07.372024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.371969    4449 scope.go:122] "RemoveContainer" containerID="9c05a6a210cbcd646920b0281a8e82733cbace7b3af3610e1a2f02ab3bff1539"
Nov 02 09:57:07.372389 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:57:07.372369    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c05a6a210cbcd646920b0281a8e82733cbace7b3af3610e1a2f02ab3bff1539\": container with ID starting with 9c05a6a210cbcd646920b0281a8e82733cbace7b3af3610e1a2f02ab3bff1539 not found: ID does not exist" containerID="9c05a6a210cbcd646920b0281a8e82733cbace7b3af3610e1a2f02ab3bff1539"
Nov 02 09:57:07.372433 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.372404    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c05a6a210cbcd646920b0281a8e82733cbace7b3af3610e1a2f02ab3bff1539"} err="failed to get container status \"9c05a6a210cbcd646920b0281a8e82733cbace7b3af3610e1a2f02ab3bff1539\": rpc error: code = NotFound desc = could not find container \"9c05a6a210cbcd646920b0281a8e82733cbace7b3af3610e1a2f02ab3bff1539\": container with ID starting with 9c05a6a210cbcd646920b0281a8e82733cbace7b3af3610e1a2f02ab3bff1539 not found: ID does not exist"
Nov 02 09:57:07.467291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.467256    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:07.467291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.467293    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:07.467509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.467308    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96v6s\" (UniqueName: \"kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-kube-api-access-96v6s\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:07.467509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.467324    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa471864-8cf5-4032-8a6d-c6c61eece6fc-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:07.663122 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.663076    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"]
Nov 02 09:57:07.675437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:07.675400    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-9a69d1-openshift-default-5c97df6d45-844jv"]
Nov 02 09:57:09.209148 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.209107    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" path="/var/lib/kubelet/pods/fa471864-8cf5-4032-8a6d-c6c61eece6fc/volumes"
Nov 02 09:57:09.619757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.619707    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"]
Nov 02 09:57:09.620731 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.620690    4449 cpu_manager.go:414] "containerMap: removing container" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy"
Nov 02 09:57:09.620731 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.620722    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy"
Nov 02 09:57:09.620966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.620768    4449 cpu_manager.go:414] "containerMap: removing container" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy"
Nov 02 09:57:09.620966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.620776    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy"
Nov 02 09:57:09.620966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.620924    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy"
Nov 02 09:57:09.626254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.626233    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:09.632704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.632679    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-dc8ee0e6e2cd\""
Nov 02 09:57:09.633946 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.633917    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"]
Nov 02 09:57:09.708618 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.708575    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-routing-overlay-volume\") pod \"payload-processing-external-model-7b5bbbbbb8-27w26\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:09.708823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.708633    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-plugins-config-volume\") pod \"payload-processing-external-model-7b5bbbbbb8-27w26\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:09.708823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.708743    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5l2b\" (UniqueName: \"kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-kube-api-access-j5l2b\") pod \"payload-processing-external-model-7b5bbbbbb8-27w26\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:09.708947 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.708830    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-provider-credentials\") pod \"payload-processing-external-model-7b5bbbbbb8-27w26\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:09.810400 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.810358    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-provider-credentials\") pod \"payload-processing-external-model-7b5bbbbbb8-27w26\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:09.810621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.810573    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-routing-overlay-volume\") pod \"payload-processing-external-model-7b5bbbbbb8-27w26\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:09.810621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.810603    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-plugins-config-volume\") pod \"payload-processing-external-model-7b5bbbbbb8-27w26\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:09.810742 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.810637    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5l2b\" (UniqueName: \"kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-kube-api-access-j5l2b\") pod \"payload-processing-external-model-7b5bbbbbb8-27w26\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:09.811290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.811260    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-routing-overlay-volume\") pod \"payload-processing-external-model-7b5bbbbbb8-27w26\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:09.811595 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.811573    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-plugins-config-volume\") pod \"payload-processing-external-model-7b5bbbbbb8-27w26\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:09.813191 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.813171    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-provider-credentials\") pod \"payload-processing-external-model-7b5bbbbbb8-27w26\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:09.823170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.823146    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5l2b\" (UniqueName: \"kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-kube-api-access-j5l2b\") pod \"payload-processing-external-model-7b5bbbbbb8-27w26\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") " pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:09.943794 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:09.943698    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:10.349312 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:10.349280    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"]
Nov 02 09:57:10.349312 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:57:10.349294    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4058fa98_f5d5_4b93_adc8_e073f39781b5.slice/crio-9baef4fb6d29443bc469659abdb5c397995d12c8cbf9c6e040cbc9754ce1b3f5 WatchSource:0}: Error finding container 9baef4fb6d29443bc469659abdb5c397995d12c8cbf9c6e040cbc9754ce1b3f5: Status 404 returned error can't find the container with id 9baef4fb6d29443bc469659abdb5c397995d12c8cbf9c6e040cbc9754ce1b3f5
Nov 02 09:57:10.366667 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:10.366637    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26" event={"ID":"4058fa98-f5d5-4b93-adc8-e073f39781b5","Type":"ContainerStarted","Data":"9baef4fb6d29443bc469659abdb5c397995d12c8cbf9c6e040cbc9754ce1b3f5"}
Nov 02 09:57:11.378631 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:11.378593    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26" event={"ID":"4058fa98-f5d5-4b93-adc8-e073f39781b5","Type":"ContainerStarted","Data":"a3d7e0de6ec61e4e9da2cfb0c7890c2d4b3aba52fdd5dd69134b67c0d4512241"}
Nov 02 09:57:11.379128 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:11.378664    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:11.402728 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:11.402669    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26" podStartSLOduration=2.4026497 podStartE2EDuration="2.4026497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:57:11.398582493 +0000 UTC m=+1308.336869296" watchObservedRunningTime="2026-11-02 09:57:11.4026497 +0000 UTC m=+1308.340936476"
Nov 02 09:57:19.715200 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.715164    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"]
Nov 02 09:57:19.716194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.716169    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa471864-8cf5-4032-8a6d-c6c61eece6fc" containerName="istio-proxy"
Nov 02 09:57:19.720038 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.720015    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"
Nov 02 09:57:19.724062 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.724028    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:57:19.724200 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.724145    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:57:19.725844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.725817    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:57:19.726016 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.725989    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:57:19.728504 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.728477    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"]
Nov 02 09:57:19.816917 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.816886    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2d9c5a2-ghl8g\" (UID: \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"
Nov 02 09:57:19.817127 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.817042    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnrk6\" (UniqueName: \"kubernetes.io/projected/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-kube-api-access-fnrk6\") pod \"maas-api-revoke-sub-b2d9c5a2-ghl8g\" (UID: \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"
Nov 02 09:57:19.817127 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.817099    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2d9c5a2-ghl8g\" (UID: \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"
Nov 02 09:57:19.918918 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.918881    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnrk6\" (UniqueName: \"kubernetes.io/projected/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-kube-api-access-fnrk6\") pod \"maas-api-revoke-sub-b2d9c5a2-ghl8g\" (UID: \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"
Nov 02 09:57:19.919085 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.918924    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2d9c5a2-ghl8g\" (UID: \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"
Nov 02 09:57:19.919131 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.919109    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2d9c5a2-ghl8g\" (UID: \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"
Nov 02 09:57:19.919786 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.919765    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2d9c5a2-ghl8g\" (UID: \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"
Nov 02 09:57:19.921435 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.921413    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2d9c5a2-ghl8g\" (UID: \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"
Nov 02 09:57:19.931088 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:19.931066    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnrk6\" (UniqueName: \"kubernetes.io/projected/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-kube-api-access-fnrk6\") pod \"maas-api-revoke-sub-b2d9c5a2-ghl8g\" (UID: \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"
Nov 02 09:57:20.042001 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:20.041972    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"
Nov 02 09:57:20.339268 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:57:20.334716    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3766ce9_16a9_41e0_adf3_8871e5bd8bc0.slice/crio-968b61f45030849bed7c45d44fcb9e88808e9ad990775c6769577cfb73a90fe3 WatchSource:0}: Error finding container 968b61f45030849bed7c45d44fcb9e88808e9ad990775c6769577cfb73a90fe3: Status 404 returned error can't find the container with id 968b61f45030849bed7c45d44fcb9e88808e9ad990775c6769577cfb73a90fe3
Nov 02 09:57:20.339268 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:20.335835    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"]
Nov 02 09:57:20.469781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:20.469746    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g" event={"ID":"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0","Type":"ContainerStarted","Data":"968b61f45030849bed7c45d44fcb9e88808e9ad990775c6769577cfb73a90fe3"}
Nov 02 09:57:21.031094 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:21.030998    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"]
Nov 02 09:57:21.037405 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:21.037365    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f77675-gnslw"]
Nov 02 09:57:21.212430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:21.212382    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13359001-d62c-44cb-bcbc-400474ff34de" path="/var/lib/kubelet/pods/13359001-d62c-44cb-bcbc-400474ff34de/volumes"
Nov 02 09:57:21.491796 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:21.491742    4449 generic.go:356] "Generic (PLEG): container finished" podID="f3766ce9-16a9-41e0-adf3-8871e5bd8bc0" containerID="960191c283185a72e94973b40b89ac092f64e76c9e4f951bbf3249dcbc764bb9" exitCode=0
Nov 02 09:57:21.492008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:21.491921    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g" event={"ID":"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0","Type":"ContainerDied","Data":"960191c283185a72e94973b40b89ac092f64e76c9e4f951bbf3249dcbc764bb9"}
Nov 02 09:57:22.391447 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:22.391411    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:22.647405 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:22.647299    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"]
Nov 02 09:57:23.234333 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:23.203447    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"
Nov 02 09:57:23.376244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:23.349738    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-xks-ca\") pod \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\" (UID: \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\") "
Nov 02 09:57:23.376244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:23.349968    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-openshift-ca\") pod \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\" (UID: \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\") "
Nov 02 09:57:23.376244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:23.350014    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-kube-api-access-fnrk6\" (UniqueName: \"kubernetes.io/projected/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-kube-api-access-fnrk6\") pod \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\" (UID: \"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0\") "
Nov 02 09:57:23.425252 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:23.419536    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-openshift-ca" pod "f3766ce9-16a9-41e0-adf3-8871e5bd8bc0" (UID: "f3766ce9-16a9-41e0-adf3-8871e5bd8bc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:57:23.641244 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:23.637691    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:23.691242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:23.668414    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-xks-ca" pod "f3766ce9-16a9-41e0-adf3-8871e5bd8bc0" (UID: "f3766ce9-16a9-41e0-adf3-8871e5bd8bc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:57:23.691242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:23.671807    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-kube-api-access-fnrk6" pod "f3766ce9-16a9-41e0-adf3-8871e5bd8bc0" (UID: "f3766ce9-16a9-41e0-adf3-8871e5bd8bc0"). InnerVolumeSpecName "kube-api-access-fnrk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:57:23.739609 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:23.739470    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:23.739609 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:23.739518    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnrk6\" (UniqueName: \"kubernetes.io/projected/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0-kube-api-access-fnrk6\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:23.828624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:23.825773    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g" event={"ID":"f3766ce9-16a9-41e0-adf3-8871e5bd8bc0","Type":"ContainerDied","Data":"968b61f45030849bed7c45d44fcb9e88808e9ad990775c6769577cfb73a90fe3"}
Nov 02 09:57:23.828624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:23.825836    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="968b61f45030849bed7c45d44fcb9e88808e9ad990775c6769577cfb73a90fe3"
Nov 02 09:57:23.828624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:23.825904    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"
Nov 02 09:57:24.632647 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:24.630446    4449 scope.go:122] "RemoveContainer" containerID="1fa7e14ed036e268a811219a7307f0c2ef55f725ecb1f92df16f64e2e9e825ba"
Nov 02 09:57:24.648005 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:24.647959    4449 scope.go:122] "RemoveContainer" containerID="fcdf85c18a5a5f8884323eb3819deed0f3b200c91c02f54ad611d23993cca080"
Nov 02 09:57:24.741596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:24.741570    4449 scope.go:122] "RemoveContainer" containerID="e2928c1c0db4e46d93446c4b0c455b61df2fd356182ebb9c194a07036663d522"
Nov 02 09:57:24.837414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:24.837380    4449 scope.go:122] "RemoveContainer" containerID="0c1d2e8b3829b57a120a6709ec71ed4e9232c61a5f8fd84f8ad47b0998e54956"
Nov 02 09:57:24.906145 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:24.906111    4449 scope.go:122] "RemoveContainer" containerID="846cb1ec527facdd56f8f117d0f91519a87aa8ab7e7fdbe4794b18a709e7daff"
Nov 02 09:57:24.928108 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:24.928083    4449 scope.go:122] "RemoveContainer" containerID="174e7a9773bc330e9069451d0499ed2fe1ddef5fd042e847e141bf0f436bf6d6"
Nov 02 09:57:24.942667 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:24.942645    4449 scope.go:122] "RemoveContainer" containerID="daac8a526e8b72699d57fcdde310babc1313661f981a857067494c545604c4ef"
Nov 02 09:57:24.953667 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:24.953541    4449 scope.go:122] "RemoveContainer" containerID="8b5bf408ff55115002b4ed8d4ad413bd68f274db61398b07b456fcce8929285a"
Nov 02 09:57:24.964157 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:24.964138    4449 scope.go:122] "RemoveContainer" containerID="e6fa159e9ac7d42a71fb507394a652c2d82c0b006c021f6651768682d477eb59"
Nov 02 09:57:27.648710 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:27.648668    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh" podUID="085247d8-11ce-41b8-9ef7-0f940cab26fd" containerName="payload-processing" containerID="cri-o://cbfb3455f6288a17e65600569ebbf59ce53cddad806d27f73a56e5003f8fec8e" gracePeriod=25
Nov 02 09:57:27.921122 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:27.921056    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:57:27.967389 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:27.967363    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"
Nov 02 09:57:27.967689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:27.967374    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh" event={"ID":"085247d8-11ce-41b8-9ef7-0f940cab26fd","Type":"ContainerDied","Data":"cbfb3455f6288a17e65600569ebbf59ce53cddad806d27f73a56e5003f8fec8e"}
Nov 02 09:57:27.967816 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:27.967797    4449 scope.go:122] "RemoveContainer" containerID="cbfb3455f6288a17e65600569ebbf59ce53cddad806d27f73a56e5003f8fec8e"
Nov 02 09:57:27.968376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:27.967204    4449 generic.go:356] "Generic (PLEG): container finished" podID="085247d8-11ce-41b8-9ef7-0f940cab26fd" containerID="cbfb3455f6288a17e65600569ebbf59ce53cddad806d27f73a56e5003f8fec8e" exitCode=0
Nov 02 09:57:27.968539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:27.968511    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-87c8cf6-sftdh" event={"ID":"085247d8-11ce-41b8-9ef7-0f940cab26fd","Type":"ContainerDied","Data":"a3168da54600c0e29521bdaac98b967318a17695587db61c9d156db2a1203aa4"}
Nov 02 09:57:27.985871 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:27.985848    4449 scope.go:122] "RemoveContainer" containerID="cbfb3455f6288a17e65600569ebbf59ce53cddad806d27f73a56e5003f8fec8e"
Nov 02 09:57:27.986170 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:57:27.986142    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbfb3455f6288a17e65600569ebbf59ce53cddad806d27f73a56e5003f8fec8e\": container with ID starting with cbfb3455f6288a17e65600569ebbf59ce53cddad806d27f73a56e5003f8fec8e not found: ID does not exist" containerID="cbfb3455f6288a17e65600569ebbf59ce53cddad806d27f73a56e5003f8fec8e"
Nov 02 09:57:27.986264 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:27.986172    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfb3455f6288a17e65600569ebbf59ce53cddad806d27f73a56e5003f8fec8e"} err="failed to get container status \"cbfb3455f6288a17e65600569ebbf59ce53cddad806d27f73a56e5003f8fec8e\": rpc error: code = NotFound desc = could not find container \"cbfb3455f6288a17e65600569ebbf59ce53cddad806d27f73a56e5003f8fec8e\": container with ID starting with cbfb3455f6288a17e65600569ebbf59ce53cddad806d27f73a56e5003f8fec8e not found: ID does not exist"
Nov 02 09:57:28.005748 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.005726    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-routing-overlay-volume\") pod \"085247d8-11ce-41b8-9ef7-0f940cab26fd\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") "
Nov 02 09:57:28.005890 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.005761    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-provider-credentials\") pod \"085247d8-11ce-41b8-9ef7-0f940cab26fd\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") "
Nov 02 09:57:28.005992 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.005976    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-kube-api-access-q45qn\" (UniqueName: \"kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-kube-api-access-q45qn\") pod \"085247d8-11ce-41b8-9ef7-0f940cab26fd\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") "
Nov 02 09:57:28.006085 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.006063    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-routing-overlay-volume" pod "085247d8-11ce-41b8-9ef7-0f940cab26fd" (UID: "085247d8-11ce-41b8-9ef7-0f940cab26fd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:57:28.006171 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.006155    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-plugins-config-volume\") pod \"085247d8-11ce-41b8-9ef7-0f940cab26fd\" (UID: \"085247d8-11ce-41b8-9ef7-0f940cab26fd\") "
Nov 02 09:57:28.006706 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.006684    4449 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-routing-overlay-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:28.006875 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.006851    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-plugins-config-volume" pod "085247d8-11ce-41b8-9ef7-0f940cab26fd" (UID: "085247d8-11ce-41b8-9ef7-0f940cab26fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:57:28.008336 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.008314    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-provider-credentials" pod "085247d8-11ce-41b8-9ef7-0f940cab26fd" (UID: "085247d8-11ce-41b8-9ef7-0f940cab26fd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:57:28.008569 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.008541    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-kube-api-access-q45qn" pod "085247d8-11ce-41b8-9ef7-0f940cab26fd" (UID: "085247d8-11ce-41b8-9ef7-0f940cab26fd"). InnerVolumeSpecName "kube-api-access-q45qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:57:28.107942 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.107901    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q45qn\" (UniqueName: \"kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-kube-api-access-q45qn\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:28.107942 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.107935    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085247d8-11ce-41b8-9ef7-0f940cab26fd-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:28.107942 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.107949    4449 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/085247d8-11ce-41b8-9ef7-0f940cab26fd-provider-credentials\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:28.310784 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.310745    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"]
Nov 02 09:57:28.320271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:28.320206    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-87c8cf6-sftdh"]
Nov 02 09:57:29.209681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:29.209645    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="085247d8-11ce-41b8-9ef7-0f940cab26fd" path="/var/lib/kubelet/pods/085247d8-11ce-41b8-9ef7-0f940cab26fd/volumes"
Nov 02 09:57:35.028940 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:35.028905    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"]
Nov 02 09:57:35.033575 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:35.033549    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba81329d-6lbzx"]
Nov 02 09:57:35.207966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:35.207932    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782c8ffc-52ff-467b-a380-cc2d4e77c952" path="/var/lib/kubelet/pods/782c8ffc-52ff-467b-a380-cc2d4e77c952/volumes"
Nov 02 09:57:36.555532 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:57:36.555489    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:57:44.937856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:44.937810    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"]
Nov 02 09:57:47.032189 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:47.032154    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"]
Nov 02 09:57:47.036995 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:47.036965    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4605dd-94gfr"]
Nov 02 09:57:47.207545 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:47.207512    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf96d9a-aa33-4614-9a3b-1f4d90ada054" path="/var/lib/kubelet/pods/cdf96d9a-aa33-4614-9a3b-1f4d90ada054/volumes"
Nov 02 09:57:49.938356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:49.938251    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26" podUID="4058fa98-f5d5-4b93-adc8-e073f39781b5" containerName="payload-processing" containerID="cri-o://a3d7e0de6ec61e4e9da2cfb0c7890c2d4b3aba52fdd5dd69134b67c0d4512241" gracePeriod=25
Nov 02 09:57:50.298915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.298875    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:50.340918 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.340879    4449 generic.go:356] "Generic (PLEG): container finished" podID="4058fa98-f5d5-4b93-adc8-e073f39781b5" containerID="a3d7e0de6ec61e4e9da2cfb0c7890c2d4b3aba52fdd5dd69134b67c0d4512241" exitCode=0
Nov 02 09:57:50.341125 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.340989    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26" event={"ID":"4058fa98-f5d5-4b93-adc8-e073f39781b5","Type":"ContainerDied","Data":"a3d7e0de6ec61e4e9da2cfb0c7890c2d4b3aba52fdd5dd69134b67c0d4512241"}
Nov 02 09:57:50.341341 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.341154    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26" event={"ID":"4058fa98-f5d5-4b93-adc8-e073f39781b5","Type":"ContainerDied","Data":"9baef4fb6d29443bc469659abdb5c397995d12c8cbf9c6e040cbc9754ce1b3f5"}
Nov 02 09:57:50.341341 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.341191    4449 scope.go:122] "RemoveContainer" containerID="a3d7e0de6ec61e4e9da2cfb0c7890c2d4b3aba52fdd5dd69134b67c0d4512241"
Nov 02 09:57:50.341691 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.341670    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"
Nov 02 09:57:50.360423 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.360394    4449 scope.go:122] "RemoveContainer" containerID="a3d7e0de6ec61e4e9da2cfb0c7890c2d4b3aba52fdd5dd69134b67c0d4512241"
Nov 02 09:57:50.360885 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:57:50.360856    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d7e0de6ec61e4e9da2cfb0c7890c2d4b3aba52fdd5dd69134b67c0d4512241\": container with ID starting with a3d7e0de6ec61e4e9da2cfb0c7890c2d4b3aba52fdd5dd69134b67c0d4512241 not found: ID does not exist" containerID="a3d7e0de6ec61e4e9da2cfb0c7890c2d4b3aba52fdd5dd69134b67c0d4512241"
Nov 02 09:57:50.360993 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.360901    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d7e0de6ec61e4e9da2cfb0c7890c2d4b3aba52fdd5dd69134b67c0d4512241"} err="failed to get container status \"a3d7e0de6ec61e4e9da2cfb0c7890c2d4b3aba52fdd5dd69134b67c0d4512241\": rpc error: code = NotFound desc = could not find container \"a3d7e0de6ec61e4e9da2cfb0c7890c2d4b3aba52fdd5dd69134b67c0d4512241\": container with ID starting with a3d7e0de6ec61e4e9da2cfb0c7890c2d4b3aba52fdd5dd69134b67c0d4512241 not found: ID does not exist"
Nov 02 09:57:50.475919 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.475884    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-kube-api-access-j5l2b\" (UniqueName: \"kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-kube-api-access-j5l2b\") pod \"4058fa98-f5d5-4b93-adc8-e073f39781b5\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") "
Nov 02 09:57:50.476129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.476006    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-provider-credentials\") pod \"4058fa98-f5d5-4b93-adc8-e073f39781b5\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") "
Nov 02 09:57:50.476129 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.476116    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-routing-overlay-volume\") pod \"4058fa98-f5d5-4b93-adc8-e073f39781b5\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") "
Nov 02 09:57:50.476551 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.476520    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-routing-overlay-volume" pod "4058fa98-f5d5-4b93-adc8-e073f39781b5" (UID: "4058fa98-f5d5-4b93-adc8-e073f39781b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:57:50.476685 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.476655    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-plugins-config-volume\") pod \"4058fa98-f5d5-4b93-adc8-e073f39781b5\" (UID: \"4058fa98-f5d5-4b93-adc8-e073f39781b5\") "
Nov 02 09:57:50.477168 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.477128    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-plugins-config-volume" pod "4058fa98-f5d5-4b93-adc8-e073f39781b5" (UID: "4058fa98-f5d5-4b93-adc8-e073f39781b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:57:50.477674 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.477649    4449 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-routing-overlay-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:50.477801 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.477678    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4058fa98-f5d5-4b93-adc8-e073f39781b5-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:50.478691 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.478664    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-kube-api-access-j5l2b" pod "4058fa98-f5d5-4b93-adc8-e073f39781b5" (UID: "4058fa98-f5d5-4b93-adc8-e073f39781b5"). InnerVolumeSpecName "kube-api-access-j5l2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:57:50.478931 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.478907    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-provider-credentials" pod "4058fa98-f5d5-4b93-adc8-e073f39781b5" (UID: "4058fa98-f5d5-4b93-adc8-e073f39781b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:57:50.579539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.579501    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5l2b\" (UniqueName: \"kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-kube-api-access-j5l2b\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:50.579539 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.579534    4449 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4058fa98-f5d5-4b93-adc8-e073f39781b5-provider-credentials\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:50.676990 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.676940    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"]
Nov 02 09:57:50.685031 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:50.684992    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5bbbbbb8-27w26"]
Nov 02 09:57:51.228873 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:51.228823    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4058fa98-f5d5-4b93-adc8-e073f39781b5" path="/var/lib/kubelet/pods/4058fa98-f5d5-4b93-adc8-e073f39781b5/volumes"
Nov 02 09:57:57.788932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.788898    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"]
Nov 02 09:57:57.789629 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.789611    4449 cpu_manager.go:414] "containerMap: removing container" podUID="085247d8-11ce-41b8-9ef7-0f940cab26fd" containerName="payload-processing"
Nov 02 09:57:57.789677 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.789630    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085247d8-11ce-41b8-9ef7-0f940cab26fd" containerName="payload-processing"
Nov 02 09:57:57.789677 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.789646    4449 cpu_manager.go:414] "containerMap: removing container" podUID="4058fa98-f5d5-4b93-adc8-e073f39781b5" containerName="payload-processing"
Nov 02 09:57:57.789677 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.789651    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4058fa98-f5d5-4b93-adc8-e073f39781b5" containerName="payload-processing"
Nov 02 09:57:57.789677 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.789674    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f3766ce9-16a9-41e0-adf3-8871e5bd8bc0" containerName="revoke-keys"
Nov 02 09:57:57.789806 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.789679    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3766ce9-16a9-41e0-adf3-8871e5bd8bc0" containerName="revoke-keys"
Nov 02 09:57:57.789806 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.789788    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3766ce9-16a9-41e0-adf3-8871e5bd8bc0" containerName="revoke-keys"
Nov 02 09:57:57.789806 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.789802    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="085247d8-11ce-41b8-9ef7-0f940cab26fd" containerName="payload-processing"
Nov 02 09:57:57.789897 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.789809    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="4058fa98-f5d5-4b93-adc8-e073f39781b5" containerName="payload-processing"
Nov 02 09:57:57.795040 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.795018    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"
Nov 02 09:57:57.799276 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.799248    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:57:57.800443 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.800354    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:57:57.800443 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.800416    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:57:57.800443 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.800433    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:57:57.803381 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.803358    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"]
Nov 02 09:57:57.918449 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.918418    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"]
Nov 02 09:57:57.923018 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.922995    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"
Nov 02 09:57:57.932204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.932015    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"]
Nov 02 09:57:57.943117 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.943089    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwpt6\" (UniqueName: \"kubernetes.io/projected/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-kube-api-access-kwpt6\") pod \"maas-api-revoke-sub-a0fd74e2-8tmxg\" (UID: \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"
Nov 02 09:57:57.943361 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.943343    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0fd74e2-8tmxg\" (UID: \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"
Nov 02 09:57:57.943427 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:57.943385    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0fd74e2-8tmxg\" (UID: \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"
Nov 02 09:57:58.045031 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.044933    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0fd74e2-8tmxg\" (UID: \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"
Nov 02 09:57:58.045031 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.045001    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0fd74e2-8tmxg\" (UID: \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"
Nov 02 09:57:58.045350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.045041    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc4208e-79hx9\" (UID: \"640c1730-9543-4fce-b997-56ca87b67dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"
Nov 02 09:57:58.045350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.045072    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwpt6\" (UniqueName: \"kubernetes.io/projected/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-kube-api-access-kwpt6\") pod \"maas-api-revoke-sub-a0fd74e2-8tmxg\" (UID: \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"
Nov 02 09:57:58.045350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.045325    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc4208e-79hx9\" (UID: \"640c1730-9543-4fce-b997-56ca87b67dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"
Nov 02 09:57:58.045530 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.045420    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhs2\" (UniqueName: \"kubernetes.io/projected/640c1730-9543-4fce-b997-56ca87b67dfe-kube-api-access-dlhs2\") pod \"maas-api-revoke-sub-bfc4208e-79hx9\" (UID: \"640c1730-9543-4fce-b997-56ca87b67dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"
Nov 02 09:57:58.045904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.045852    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0fd74e2-8tmxg\" (UID: \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"
Nov 02 09:57:58.047696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.047675    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0fd74e2-8tmxg\" (UID: \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"
Nov 02 09:57:58.058920 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.058885    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwpt6\" (UniqueName: \"kubernetes.io/projected/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-kube-api-access-kwpt6\") pod \"maas-api-revoke-sub-a0fd74e2-8tmxg\" (UID: \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"
Nov 02 09:57:58.114166 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.114130    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"
Nov 02 09:57:58.146631 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.146587    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc4208e-79hx9\" (UID: \"640c1730-9543-4fce-b997-56ca87b67dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"
Nov 02 09:57:58.146816 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.146666    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc4208e-79hx9\" (UID: \"640c1730-9543-4fce-b997-56ca87b67dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"
Nov 02 09:57:58.146816 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.146734    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhs2\" (UniqueName: \"kubernetes.io/projected/640c1730-9543-4fce-b997-56ca87b67dfe-kube-api-access-dlhs2\") pod \"maas-api-revoke-sub-bfc4208e-79hx9\" (UID: \"640c1730-9543-4fce-b997-56ca87b67dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"
Nov 02 09:57:58.147618 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.147587    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc4208e-79hx9\" (UID: \"640c1730-9543-4fce-b997-56ca87b67dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"
Nov 02 09:57:58.150605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.150576    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc4208e-79hx9\" (UID: \"640c1730-9543-4fce-b997-56ca87b67dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"
Nov 02 09:57:58.163769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.162976    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhs2\" (UniqueName: \"kubernetes.io/projected/640c1730-9543-4fce-b997-56ca87b67dfe-kube-api-access-dlhs2\") pod \"maas-api-revoke-sub-bfc4208e-79hx9\" (UID: \"640c1730-9543-4fce-b997-56ca87b67dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"
Nov 02 09:57:58.241025 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.240979    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"
Nov 02 09:57:58.410861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.410580    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"]
Nov 02 09:57:58.411123 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:57:58.411099    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd14e18c_0c04_4e1d_90b2_c935c7e9cdfd.slice/crio-8b5645f29bf7d878705b5f2f4f42ca32655dba05eeb8ab70a42aba2bafb175c8 WatchSource:0}: Error finding container 8b5645f29bf7d878705b5f2f4f42ca32655dba05eeb8ab70a42aba2bafb175c8: Status 404 returned error can't find the container with id 8b5645f29bf7d878705b5f2f4f42ca32655dba05eeb8ab70a42aba2bafb175c8
Nov 02 09:57:58.477919 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.477883    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg" event={"ID":"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd","Type":"ContainerStarted","Data":"8b5645f29bf7d878705b5f2f4f42ca32655dba05eeb8ab70a42aba2bafb175c8"}
Nov 02 09:57:58.530624 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:57:58.530590    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640c1730_9543_4fce_b997_56ca87b67dfe.slice/crio-ea542fd601546a0d8d3bf59ef4a031d934288f6c3ddd33f73bf2f669e8230d08 WatchSource:0}: Error finding container ea542fd601546a0d8d3bf59ef4a031d934288f6c3ddd33f73bf2f669e8230d08: Status 404 returned error can't find the container with id ea542fd601546a0d8d3bf59ef4a031d934288f6c3ddd33f73bf2f669e8230d08
Nov 02 09:57:58.531021 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:58.530998    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"]
Nov 02 09:57:59.490541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:59.490447    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg" event={"ID":"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd","Type":"ContainerStarted","Data":"006c97ce6e255ad126ea9d51bd680ca42f4f44e8eaed274a65ef5eeeef7b019e"}
Nov 02 09:57:59.492553 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:59.492525    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9" event={"ID":"640c1730-9543-4fce-b997-56ca87b67dfe","Type":"ContainerStarted","Data":"2581debc8b07cafe64a9d234d8d3e9f47df42c554d32715a23200d02d52bab98"}
Nov 02 09:57:59.492814 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:59.492754    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9" event={"ID":"640c1730-9543-4fce-b997-56ca87b67dfe","Type":"ContainerStarted","Data":"ea542fd601546a0d8d3bf59ef4a031d934288f6c3ddd33f73bf2f669e8230d08"}
Nov 02 09:57:59.513403 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:59.513351    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg" podStartSLOduration=2.513335922 podStartE2EDuration="2.513335922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:57:59.507919766 +0000 UTC m=+1356.446206543" watchObservedRunningTime="2026-11-02 09:57:59.513335922 +0000 UTC m=+1356.451622698"
Nov 02 09:57:59.529455 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:57:59.529408    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9" podStartSLOduration=2.52939664 podStartE2EDuration="2.52939664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:57:59.525314556 +0000 UTC m=+1356.463601357" watchObservedRunningTime="2026-11-02 09:57:59.52939664 +0000 UTC m=+1356.467683416"
Nov 02 09:58:00.504398 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:00.504358    4449 generic.go:356] "Generic (PLEG): container finished" podID="640c1730-9543-4fce-b997-56ca87b67dfe" containerID="2581debc8b07cafe64a9d234d8d3e9f47df42c554d32715a23200d02d52bab98" exitCode=0
Nov 02 09:58:00.504833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:00.504443    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9" event={"ID":"640c1730-9543-4fce-b997-56ca87b67dfe","Type":"ContainerDied","Data":"2581debc8b07cafe64a9d234d8d3e9f47df42c554d32715a23200d02d52bab98"}
Nov 02 09:58:00.506297 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:00.506275    4449 generic.go:356] "Generic (PLEG): container finished" podID="cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd" containerID="006c97ce6e255ad126ea9d51bd680ca42f4f44e8eaed274a65ef5eeeef7b019e" exitCode=0
Nov 02 09:58:00.506433 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:00.506361    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg" event={"ID":"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd","Type":"ContainerDied","Data":"006c97ce6e255ad126ea9d51bd680ca42f4f44e8eaed274a65ef5eeeef7b019e"}
Nov 02 09:58:01.028042 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.028003    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"]
Nov 02 09:58:01.032646 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.032612    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1bd7cc1-94rdh"]
Nov 02 09:58:01.208283 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.208207    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d02db6-2385-4eb2-98ce-fae2c070af0c" path="/var/lib/kubelet/pods/b4d02db6-2385-4eb2-98ce-fae2c070af0c/volumes"
Nov 02 09:58:01.728967 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.728932    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"
Nov 02 09:58:01.755757 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.755730    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"
Nov 02 09:58:01.819975 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.819907    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-openshift-ca\") pod \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\" (UID: \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\") "
Nov 02 09:58:01.820306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.820014    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-xks-ca\") pod \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\" (UID: \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\") "
Nov 02 09:58:01.820306 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.820068    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-kube-api-access-kwpt6\" (UniqueName: \"kubernetes.io/projected/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-kube-api-access-kwpt6\") pod \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\" (UID: \"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd\") "
Nov 02 09:58:01.820624 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.820599    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-openshift-ca" pod "cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd" (UID: "cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:01.820943 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.820906    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:01.825622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.824741    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-kube-api-access-kwpt6" pod "cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd" (UID: "cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd"). InnerVolumeSpecName "kube-api-access-kwpt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:01.826915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.826891    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-xks-ca" pod "cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd" (UID: "cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:58:01.921743 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.921708    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/640c1730-9543-4fce-b997-56ca87b67dfe-kube-api-access-dlhs2\" (UniqueName: \"kubernetes.io/projected/640c1730-9543-4fce-b997-56ca87b67dfe-kube-api-access-dlhs2\") pod \"640c1730-9543-4fce-b997-56ca87b67dfe\" (UID: \"640c1730-9543-4fce-b997-56ca87b67dfe\") "
Nov 02 09:58:01.921937 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.921768    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-xks-ca\") pod \"640c1730-9543-4fce-b997-56ca87b67dfe\" (UID: \"640c1730-9543-4fce-b997-56ca87b67dfe\") "
Nov 02 09:58:01.921937 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.921792    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-openshift-ca\") pod \"640c1730-9543-4fce-b997-56ca87b67dfe\" (UID: \"640c1730-9543-4fce-b997-56ca87b67dfe\") "
Nov 02 09:58:01.922271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.922246    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-openshift-ca" pod "640c1730-9543-4fce-b997-56ca87b67dfe" (UID: "640c1730-9543-4fce-b997-56ca87b67dfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:01.922352 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.922308    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:01.922352 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.922326    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwpt6\" (UniqueName: \"kubernetes.io/projected/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd-kube-api-access-kwpt6\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:01.924089 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.924068    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-xks-ca" pod "640c1730-9543-4fce-b997-56ca87b67dfe" (UID: "640c1730-9543-4fce-b997-56ca87b67dfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:58:01.924132 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:01.924111    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640c1730-9543-4fce-b997-56ca87b67dfe-kube-api-access-dlhs2" pod "640c1730-9543-4fce-b997-56ca87b67dfe" (UID: "640c1730-9543-4fce-b997-56ca87b67dfe"). InnerVolumeSpecName "kube-api-access-dlhs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:02.024303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:02.024267    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlhs2\" (UniqueName: \"kubernetes.io/projected/640c1730-9543-4fce-b997-56ca87b67dfe-kube-api-access-dlhs2\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:02.024303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:02.024301    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:02.024303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:02.024311    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/640c1730-9543-4fce-b997-56ca87b67dfe-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:02.351293 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:02.351259    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"]
Nov 02 09:58:02.524837 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:02.524807    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"
Nov 02 09:58:02.525030 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:02.524844    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg" event={"ID":"cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd","Type":"ContainerDied","Data":"8b5645f29bf7d878705b5f2f4f42ca32655dba05eeb8ab70a42aba2bafb175c8"}
Nov 02 09:58:02.525030 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:02.524938    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b5645f29bf7d878705b5f2f4f42ca32655dba05eeb8ab70a42aba2bafb175c8"
Nov 02 09:58:02.526559 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:02.526532    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"
Nov 02 09:58:02.526672 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:02.526576    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9" event={"ID":"640c1730-9543-4fce-b997-56ca87b67dfe","Type":"ContainerDied","Data":"ea542fd601546a0d8d3bf59ef4a031d934288f6c3ddd33f73bf2f669e8230d08"}
Nov 02 09:58:02.526672 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:02.526622    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea542fd601546a0d8d3bf59ef4a031d934288f6c3ddd33f73bf2f669e8230d08"
Nov 02 09:58:06.556550 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:06.556398    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:58:08.597635 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:08.597606    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"]
Nov 02 09:58:08.612307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:08.612270    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"]
Nov 02 09:58:10.019010 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.018962    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"]
Nov 02 09:58:10.019859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.019834    4449 cpu_manager.go:414] "containerMap: removing container" podUID="640c1730-9543-4fce-b997-56ca87b67dfe" containerName="revoke-keys"
Nov 02 09:58:10.019977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.019859    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640c1730-9543-4fce-b997-56ca87b67dfe" containerName="revoke-keys"
Nov 02 09:58:10.019977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.019909    4449 cpu_manager.go:414] "containerMap: removing container" podUID="cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd" containerName="revoke-keys"
Nov 02 09:58:10.019977 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.019918    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd" containerName="revoke-keys"
Nov 02 09:58:10.020134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.020098    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="640c1730-9543-4fce-b997-56ca87b67dfe" containerName="revoke-keys"
Nov 02 09:58:10.020134 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.020118    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd" containerName="revoke-keys"
Nov 02 09:58:10.024140 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.024118    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"
Nov 02 09:58:10.028448 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.028404    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:58:10.030438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.029687    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:58:10.030438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.029718    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:58:10.030438 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.029782    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:58:10.032663 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.032640    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"]
Nov 02 09:58:10.056034 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.055994    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-775343e0-vxqj7\" (UID: \"e23a5653-492b-4f80-b6ca-f32b35c49582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"
Nov 02 09:58:10.056286 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.056055    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-775343e0-vxqj7\" (UID: \"e23a5653-492b-4f80-b6ca-f32b35c49582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"
Nov 02 09:58:10.056286 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.056265    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsw4d\" (UniqueName: \"kubernetes.io/projected/e23a5653-492b-4f80-b6ca-f32b35c49582-kube-api-access-wsw4d\") pod \"maas-api-revoke-sub-775343e0-vxqj7\" (UID: \"e23a5653-492b-4f80-b6ca-f32b35c49582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"
Nov 02 09:58:10.157785 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.157754    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-775343e0-vxqj7\" (UID: \"e23a5653-492b-4f80-b6ca-f32b35c49582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"
Nov 02 09:58:10.157961 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.157807    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-775343e0-vxqj7\" (UID: \"e23a5653-492b-4f80-b6ca-f32b35c49582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"
Nov 02 09:58:10.157961 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.157861    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsw4d\" (UniqueName: \"kubernetes.io/projected/e23a5653-492b-4f80-b6ca-f32b35c49582-kube-api-access-wsw4d\") pod \"maas-api-revoke-sub-775343e0-vxqj7\" (UID: \"e23a5653-492b-4f80-b6ca-f32b35c49582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"
Nov 02 09:58:10.158603 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.158532    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-775343e0-vxqj7\" (UID: \"e23a5653-492b-4f80-b6ca-f32b35c49582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"
Nov 02 09:58:10.160351 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.160330    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-775343e0-vxqj7\" (UID: \"e23a5653-492b-4f80-b6ca-f32b35c49582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"
Nov 02 09:58:10.170363 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.170331    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsw4d\" (UniqueName: \"kubernetes.io/projected/e23a5653-492b-4f80-b6ca-f32b35c49582-kube-api-access-wsw4d\") pod \"maas-api-revoke-sub-775343e0-vxqj7\" (UID: \"e23a5653-492b-4f80-b6ca-f32b35c49582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"
Nov 02 09:58:10.267045 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.267012    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"]
Nov 02 09:58:10.271953 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.271886    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"
Nov 02 09:58:10.280910 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.280877    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"]
Nov 02 09:58:10.340145 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.340115    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"
Nov 02 09:58:10.364571 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.364532    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f8ebf2-qdgl8\" (UID: \"ac8f37f0-1a7c-404b-a032-c7b221388227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"
Nov 02 09:58:10.366232 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.366173    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f8ebf2-qdgl8\" (UID: \"ac8f37f0-1a7c-404b-a032-c7b221388227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"
Nov 02 09:58:10.366670 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.366644    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx7kb\" (UniqueName: \"kubernetes.io/projected/ac8f37f0-1a7c-404b-a032-c7b221388227-kube-api-access-dx7kb\") pod \"maas-api-revoke-sub-47f8ebf2-qdgl8\" (UID: \"ac8f37f0-1a7c-404b-a032-c7b221388227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"
Nov 02 09:58:10.468610 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.467958    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f8ebf2-qdgl8\" (UID: \"ac8f37f0-1a7c-404b-a032-c7b221388227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"
Nov 02 09:58:10.468610 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.468034    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx7kb\" (UniqueName: \"kubernetes.io/projected/ac8f37f0-1a7c-404b-a032-c7b221388227-kube-api-access-dx7kb\") pod \"maas-api-revoke-sub-47f8ebf2-qdgl8\" (UID: \"ac8f37f0-1a7c-404b-a032-c7b221388227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"
Nov 02 09:58:10.468610 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.468086    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f8ebf2-qdgl8\" (UID: \"ac8f37f0-1a7c-404b-a032-c7b221388227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"
Nov 02 09:58:10.468959 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.468910    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f8ebf2-qdgl8\" (UID: \"ac8f37f0-1a7c-404b-a032-c7b221388227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"
Nov 02 09:58:10.471133 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.471101    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f8ebf2-qdgl8\" (UID: \"ac8f37f0-1a7c-404b-a032-c7b221388227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"
Nov 02 09:58:10.481338 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.481310    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx7kb\" (UniqueName: \"kubernetes.io/projected/ac8f37f0-1a7c-404b-a032-c7b221388227-kube-api-access-dx7kb\") pod \"maas-api-revoke-sub-47f8ebf2-qdgl8\" (UID: \"ac8f37f0-1a7c-404b-a032-c7b221388227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"
Nov 02 09:58:10.593201 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.593163    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"
Nov 02 09:58:10.646042 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.645956    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"]
Nov 02 09:58:10.646208 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:58:10.646157    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23a5653_492b_4f80_b6ca_f32b35c49582.slice/crio-e175f97a3b4aeb125dc83b8e5825de293e02c4f037c6a93a9b497f54b4108f6e WatchSource:0}: Error finding container e175f97a3b4aeb125dc83b8e5825de293e02c4f037c6a93a9b497f54b4108f6e: Status 404 returned error can't find the container with id e175f97a3b4aeb125dc83b8e5825de293e02c4f037c6a93a9b497f54b4108f6e
Nov 02 09:58:10.884968 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:58:10.884939    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8f37f0_1a7c_404b_a032_c7b221388227.slice/crio-e59d2a29170b7727f035d4c48f5b9055a55fc3a5dbc9542115d81d15f8420023 WatchSource:0}: Error finding container e59d2a29170b7727f035d4c48f5b9055a55fc3a5dbc9542115d81d15f8420023: Status 404 returned error can't find the container with id e59d2a29170b7727f035d4c48f5b9055a55fc3a5dbc9542115d81d15f8420023
Nov 02 09:58:10.886406 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:10.886383    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"]
Nov 02 09:58:11.027942 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:11.027907    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"]
Nov 02 09:58:11.032447 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:11.032421    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc0d550-dpmqz"]
Nov 02 09:58:11.207422 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:11.207387    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d9310b-570a-41d6-9eb3-54dd7aab79bf" path="/var/lib/kubelet/pods/d5d9310b-570a-41d6-9eb3-54dd7aab79bf/volumes"
Nov 02 09:58:11.615616 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:11.615573    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7" event={"ID":"e23a5653-492b-4f80-b6ca-f32b35c49582","Type":"ContainerStarted","Data":"849d183ac2b232a7c3bebe2b33e67c1e8141f050ae7ef38d7ed6620123f2d23e"}
Nov 02 09:58:11.615820 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:11.615635    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7" event={"ID":"e23a5653-492b-4f80-b6ca-f32b35c49582","Type":"ContainerStarted","Data":"e175f97a3b4aeb125dc83b8e5825de293e02c4f037c6a93a9b497f54b4108f6e"}
Nov 02 09:58:11.617563 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:11.617531    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8" event={"ID":"ac8f37f0-1a7c-404b-a032-c7b221388227","Type":"ContainerStarted","Data":"97255e58a65324de0772bc0ef27ea4fdfc76cce5f36eab51ac96d38ce8af2bc9"}
Nov 02 09:58:11.617690 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:11.617588    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8" event={"ID":"ac8f37f0-1a7c-404b-a032-c7b221388227","Type":"ContainerStarted","Data":"e59d2a29170b7727f035d4c48f5b9055a55fc3a5dbc9542115d81d15f8420023"}
Nov 02 09:58:11.634723 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:11.634674    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7" podStartSLOduration=2.634660705 podStartE2EDuration="2.634660705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:58:11.633393441 +0000 UTC m=+1368.571680238" watchObservedRunningTime="2026-11-02 09:58:11.634660705 +0000 UTC m=+1368.572947480"
Nov 02 09:58:11.657275 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:11.657187    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8" podStartSLOduration=1.657167598 podStartE2EDuration="1.657167598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:58:11.651071387 +0000 UTC m=+1368.589358186" watchObservedRunningTime="2026-11-02 09:58:11.657167598 +0000 UTC m=+1368.595454375"
Nov 02 09:58:12.628029 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:12.627995    4449 generic.go:356] "Generic (PLEG): container finished" podID="e23a5653-492b-4f80-b6ca-f32b35c49582" containerID="849d183ac2b232a7c3bebe2b33e67c1e8141f050ae7ef38d7ed6620123f2d23e" exitCode=0
Nov 02 09:58:12.628576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:12.628084    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7" event={"ID":"e23a5653-492b-4f80-b6ca-f32b35c49582","Type":"ContainerDied","Data":"849d183ac2b232a7c3bebe2b33e67c1e8141f050ae7ef38d7ed6620123f2d23e"}
Nov 02 09:58:12.630012 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:12.629987    4449 generic.go:356] "Generic (PLEG): container finished" podID="ac8f37f0-1a7c-404b-a032-c7b221388227" containerID="97255e58a65324de0772bc0ef27ea4fdfc76cce5f36eab51ac96d38ce8af2bc9" exitCode=0
Nov 02 09:58:12.630147 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:12.630076    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8" event={"ID":"ac8f37f0-1a7c-404b-a032-c7b221388227","Type":"ContainerDied","Data":"97255e58a65324de0772bc0ef27ea4fdfc76cce5f36eab51ac96d38ce8af2bc9"}
Nov 02 09:58:13.598331 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:13.598285    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9" podUID="c2d7f4ec-14a0-450a-9298-c6da1c6fba89" containerName="payload-processing" containerID="cri-o://4067e0133864cc21ce45e35d3dd3338699b9d139773c83f2fd32969045d00a63" gracePeriod=25
Nov 02 09:58:13.613251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:13.613149    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j" podUID="eec88150-3595-4245-a745-3e5cb6e0455d" containerName="payload-pre-processing" containerID="cri-o://68196c16ea44175392cb383680a792ccef636a0653f024e93fb769a948314916" gracePeriod=25
Nov 02 09:58:14.247995 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.247113    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"
Nov 02 09:58:14.541966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.540921    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8f37f0-1a7c-404b-a032-c7b221388227-kube-api-access-dx7kb\" (UniqueName: \"kubernetes.io/projected/ac8f37f0-1a7c-404b-a032-c7b221388227-kube-api-access-dx7kb\") pod \"ac8f37f0-1a7c-404b-a032-c7b221388227\" (UID: \"ac8f37f0-1a7c-404b-a032-c7b221388227\") "
Nov 02 09:58:14.541966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.541244    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-openshift-ca\") pod \"ac8f37f0-1a7c-404b-a032-c7b221388227\" (UID: \"ac8f37f0-1a7c-404b-a032-c7b221388227\") "
Nov 02 09:58:14.541966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.541338    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-xks-ca\") pod \"ac8f37f0-1a7c-404b-a032-c7b221388227\" (UID: \"ac8f37f0-1a7c-404b-a032-c7b221388227\") "
Nov 02 09:58:14.578818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.545069    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-openshift-ca" pod "ac8f37f0-1a7c-404b-a032-c7b221388227" (UID: "ac8f37f0-1a7c-404b-a032-c7b221388227"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:14.582598 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.581156    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-xks-ca" pod "ac8f37f0-1a7c-404b-a032-c7b221388227" (UID: "ac8f37f0-1a7c-404b-a032-c7b221388227"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:58:14.587400 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.587047    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8f37f0-1a7c-404b-a032-c7b221388227-kube-api-access-dx7kb" pod "ac8f37f0-1a7c-404b-a032-c7b221388227" (UID: "ac8f37f0-1a7c-404b-a032-c7b221388227"). InnerVolumeSpecName "kube-api-access-dx7kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:14.636669 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.636645    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"
Nov 02 09:58:14.647859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.645664    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:14.647859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.645703    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8f37f0-1a7c-404b-a032-c7b221388227-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:14.647859 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.645722    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx7kb\" (UniqueName: \"kubernetes.io/projected/ac8f37f0-1a7c-404b-a032-c7b221388227-kube-api-access-dx7kb\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:14.746473 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.746377    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-openshift-ca\") pod \"e23a5653-492b-4f80-b6ca-f32b35c49582\" (UID: \"e23a5653-492b-4f80-b6ca-f32b35c49582\") "
Nov 02 09:58:14.747104 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.747080    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23a5653-492b-4f80-b6ca-f32b35c49582-kube-api-access-wsw4d\" (UniqueName: \"kubernetes.io/projected/e23a5653-492b-4f80-b6ca-f32b35c49582-kube-api-access-wsw4d\") pod \"e23a5653-492b-4f80-b6ca-f32b35c49582\" (UID: \"e23a5653-492b-4f80-b6ca-f32b35c49582\") "
Nov 02 09:58:14.747385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.747366    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-xks-ca\") pod \"e23a5653-492b-4f80-b6ca-f32b35c49582\" (UID: \"e23a5653-492b-4f80-b6ca-f32b35c49582\") "
Nov 02 09:58:14.749351 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.749308    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-openshift-ca" pod "e23a5653-492b-4f80-b6ca-f32b35c49582" (UID: "e23a5653-492b-4f80-b6ca-f32b35c49582"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:14.749794 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.749765    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7" event={"ID":"e23a5653-492b-4f80-b6ca-f32b35c49582","Type":"ContainerDied","Data":"e175f97a3b4aeb125dc83b8e5825de293e02c4f037c6a93a9b497f54b4108f6e"}
Nov 02 09:58:14.749986 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.749969    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e175f97a3b4aeb125dc83b8e5825de293e02c4f037c6a93a9b497f54b4108f6e"
Nov 02 09:58:14.750818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.750330    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"
Nov 02 09:58:14.761666 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.761628    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8" event={"ID":"ac8f37f0-1a7c-404b-a032-c7b221388227","Type":"ContainerDied","Data":"e59d2a29170b7727f035d4c48f5b9055a55fc3a5dbc9542115d81d15f8420023"}
Nov 02 09:58:14.763296 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.762059    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"
Nov 02 09:58:14.779289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.762130    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e59d2a29170b7727f035d4c48f5b9055a55fc3a5dbc9542115d81d15f8420023"
Nov 02 09:58:14.797158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.797121    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-xks-ca" pod "e23a5653-492b-4f80-b6ca-f32b35c49582" (UID: "e23a5653-492b-4f80-b6ca-f32b35c49582"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:58:14.849350 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.849318    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:14.849547 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.849534    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23a5653-492b-4f80-b6ca-f32b35c49582-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:14.857823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.857785    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23a5653-492b-4f80-b6ca-f32b35c49582-kube-api-access-wsw4d" pod "e23a5653-492b-4f80-b6ca-f32b35c49582" (UID: "e23a5653-492b-4f80-b6ca-f32b35c49582"). InnerVolumeSpecName "kube-api-access-wsw4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:14.913319 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:14.913190    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8f37f0_1a7c_404b_a032_c7b221388227.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 09:58:14.952319 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:14.951130    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsw4d\" (UniqueName: \"kubernetes.io/projected/e23a5653-492b-4f80-b6ca-f32b35c49582-kube-api-access-wsw4d\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:15.173471 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:15.173428    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9" podUID="c2d7f4ec-14a0-450a-9298-c6da1c6fba89" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:58:16.181382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:16.181340    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j" podUID="eec88150-3595-4245-a745-3e5cb6e0455d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:58:17.377597 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.377547    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main" containerID="cri-o://a9fc3710565f6b71f2d5a53c180169afe6c47eb2e10d5c73e9044ea33267546f" gracePeriod=45
Nov 02 09:58:17.697762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.697728    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:58:17.791458 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.791425    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45450305-a85b-470b-b163-b4a2d35a3cd0-tls-certs\" (UniqueName: \"kubernetes.io/secret/45450305-a85b-470b-b163-b4a2d35a3cd0-tls-certs\") pod \"45450305-a85b-470b-b163-b4a2d35a3cd0\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") "
Nov 02 09:58:17.791782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.791548    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-home\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-home\") pod \"45450305-a85b-470b-b163-b4a2d35a3cd0\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") "
Nov 02 09:58:17.791782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.791702    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-tmp-dir\") pod \"45450305-a85b-470b-b163-b4a2d35a3cd0\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") "
Nov 02 09:58:17.792020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.791820    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45450305-a85b-470b-b163-b4a2d35a3cd0-kube-api-access-h5l57\" (UniqueName: \"kubernetes.io/projected/45450305-a85b-470b-b163-b4a2d35a3cd0-kube-api-access-h5l57\") pod \"45450305-a85b-470b-b163-b4a2d35a3cd0\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") "
Nov 02 09:58:17.792020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.791853    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-home" pod "45450305-a85b-470b-b163-b4a2d35a3cd0" (UID: "45450305-a85b-470b-b163-b4a2d35a3cd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.792020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.791872    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-dshm\") pod \"45450305-a85b-470b-b163-b4a2d35a3cd0\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") "
Nov 02 09:58:17.792020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.791935    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-model-cache\") pod \"45450305-a85b-470b-b163-b4a2d35a3cd0\" (UID: \"45450305-a85b-470b-b163-b4a2d35a3cd0\") "
Nov 02 09:58:17.792020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.792010    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-tmp-dir" pod "45450305-a85b-470b-b163-b4a2d35a3cd0" (UID: "45450305-a85b-470b-b163-b4a2d35a3cd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.792371 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.792347    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-model-cache" pod "45450305-a85b-470b-b163-b4a2d35a3cd0" (UID: "45450305-a85b-470b-b163-b4a2d35a3cd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.792698 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.792680    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.792772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.792703    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.792772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.792717    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.793152 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.793125    4449 generic.go:356] "Generic (PLEG): container finished" podID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerID="a9fc3710565f6b71f2d5a53c180169afe6c47eb2e10d5c73e9044ea33267546f" exitCode=0
Nov 02 09:58:17.793281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.793190    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"
Nov 02 09:58:17.793281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.793227    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" event={"ID":"45450305-a85b-470b-b163-b4a2d35a3cd0","Type":"ContainerDied","Data":"a9fc3710565f6b71f2d5a53c180169afe6c47eb2e10d5c73e9044ea33267546f"}
Nov 02 09:58:17.793356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.793287    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7" event={"ID":"45450305-a85b-470b-b163-b4a2d35a3cd0","Type":"ContainerDied","Data":"3483b76339fe787b1469c7e9e20eaa875376598e4a1d0ddb250abcad1fb690c1"}
Nov 02 09:58:17.793356 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.793312    4449 scope.go:122] "RemoveContainer" containerID="a9fc3710565f6b71f2d5a53c180169afe6c47eb2e10d5c73e9044ea33267546f"
Nov 02 09:58:17.794806 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.794502    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45450305-a85b-470b-b163-b4a2d35a3cd0-tls-certs" pod "45450305-a85b-470b-b163-b4a2d35a3cd0" (UID: "45450305-a85b-470b-b163-b4a2d35a3cd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:58:17.794806 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.794527    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-dshm" pod "45450305-a85b-470b-b163-b4a2d35a3cd0" (UID: "45450305-a85b-470b-b163-b4a2d35a3cd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.795918 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.795894    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45450305-a85b-470b-b163-b4a2d35a3cd0-kube-api-access-h5l57" pod "45450305-a85b-470b-b163-b4a2d35a3cd0" (UID: "45450305-a85b-470b-b163-b4a2d35a3cd0"). InnerVolumeSpecName "kube-api-access-h5l57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:17.833681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.833649    4449 scope.go:122] "RemoveContainer" containerID="a9fc3710565f6b71f2d5a53c180169afe6c47eb2e10d5c73e9044ea33267546f"
Nov 02 09:58:17.834078 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:17.834044    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9fc3710565f6b71f2d5a53c180169afe6c47eb2e10d5c73e9044ea33267546f\": container with ID starting with a9fc3710565f6b71f2d5a53c180169afe6c47eb2e10d5c73e9044ea33267546f not found: ID does not exist" containerID="a9fc3710565f6b71f2d5a53c180169afe6c47eb2e10d5c73e9044ea33267546f"
Nov 02 09:58:17.834182 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.834076    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9fc3710565f6b71f2d5a53c180169afe6c47eb2e10d5c73e9044ea33267546f"} err="failed to get container status \"a9fc3710565f6b71f2d5a53c180169afe6c47eb2e10d5c73e9044ea33267546f\": rpc error: code = NotFound desc = could not find container \"a9fc3710565f6b71f2d5a53c180169afe6c47eb2e10d5c73e9044ea33267546f\": container with ID starting with a9fc3710565f6b71f2d5a53c180169afe6c47eb2e10d5c73e9044ea33267546f not found: ID does not exist"
Nov 02 09:58:17.894144 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.894101    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45450305-a85b-470b-b163-b4a2d35a3cd0-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.894144 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.894143    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5l57\" (UniqueName: \"kubernetes.io/projected/45450305-a85b-470b-b163-b4a2d35a3cd0-kube-api-access-h5l57\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.894404 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:17.894159    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45450305-a85b-470b-b163-b4a2d35a3cd0-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:18.026481 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:18.026388    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"]
Nov 02 09:58:18.031224 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:18.031187    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5740cd2-rwrj8"]
Nov 02 09:58:18.120861 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:18.120819    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"]
Nov 02 09:58:18.125516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:18.125480    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/distinct-2-w0-2e80fb-kserve-cfb5c9bb7-d2lj7"]
Nov 02 09:58:19.216616 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:19.216570    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" path="/var/lib/kubelet/pods/45450305-a85b-470b-b163-b4a2d35a3cd0/volumes"
Nov 02 09:58:19.217395 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:19.217368    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7482ee0-db9f-4b9c-bb10-3eb75511d765" path="/var/lib/kubelet/pods/a7482ee0-db9f-4b9c-bb10-3eb75511d765/volumes"
Nov 02 09:58:22.550242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.536701    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"]
Nov 02 09:58:22.550242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.537793    4449 cpu_manager.go:414] "containerMap: removing container" podUID="ac8f37f0-1a7c-404b-a032-c7b221388227" containerName="revoke-keys"
Nov 02 09:58:22.550242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.537837    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8f37f0-1a7c-404b-a032-c7b221388227" containerName="revoke-keys"
Nov 02 09:58:22.550242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.537857    4449 cpu_manager.go:414] "containerMap: removing container" podUID="e23a5653-492b-4f80-b6ca-f32b35c49582" containerName="revoke-keys"
Nov 02 09:58:22.550242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.537865    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23a5653-492b-4f80-b6ca-f32b35c49582" containerName="revoke-keys"
Nov 02 09:58:22.550242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.537874    4449 cpu_manager.go:414] "containerMap: removing container" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main"
Nov 02 09:58:22.550242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.537880    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main"
Nov 02 09:58:22.550242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.538065    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8f37f0-1a7c-404b-a032-c7b221388227" containerName="revoke-keys"
Nov 02 09:58:22.550242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.538079    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23a5653-492b-4f80-b6ca-f32b35c49582" containerName="revoke-keys"
Nov 02 09:58:22.550242 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.538088    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="45450305-a85b-470b-b163-b4a2d35a3cd0" containerName="main"
Nov 02 09:58:22.670309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.668927    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"
Nov 02 09:58:22.670723 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.668825    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"]
Nov 02 09:58:22.673501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.673468    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:58:22.673767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.673748    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:58:22.714369 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.701415    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:58:22.714369 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.701551    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:58:22.804664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.804575    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b828eafc-kt44t\" (UID: \"eef68126-35a7-4716-a359-5b31ff36b0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"
Nov 02 09:58:22.804824 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.804711    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bh5p\" (UniqueName: \"kubernetes.io/projected/eef68126-35a7-4716-a359-5b31ff36b0cf-kube-api-access-5bh5p\") pod \"maas-api-revoke-sub-b828eafc-kt44t\" (UID: \"eef68126-35a7-4716-a359-5b31ff36b0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"
Nov 02 09:58:22.804903 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.804821    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b828eafc-kt44t\" (UID: \"eef68126-35a7-4716-a359-5b31ff36b0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"
Nov 02 09:58:22.918245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.917357    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bh5p\" (UniqueName: \"kubernetes.io/projected/eef68126-35a7-4716-a359-5b31ff36b0cf-kube-api-access-5bh5p\") pod \"maas-api-revoke-sub-b828eafc-kt44t\" (UID: \"eef68126-35a7-4716-a359-5b31ff36b0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"
Nov 02 09:58:22.918245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.917530    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b828eafc-kt44t\" (UID: \"eef68126-35a7-4716-a359-5b31ff36b0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"
Nov 02 09:58:22.918245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.917697    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b828eafc-kt44t\" (UID: \"eef68126-35a7-4716-a359-5b31ff36b0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"
Nov 02 09:58:22.918952 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.918777    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b828eafc-kt44t\" (UID: \"eef68126-35a7-4716-a359-5b31ff36b0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"
Nov 02 09:58:22.935969 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.935937    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b828eafc-kt44t\" (UID: \"eef68126-35a7-4716-a359-5b31ff36b0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"
Nov 02 09:58:22.938660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:22.938594    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bh5p\" (UniqueName: \"kubernetes.io/projected/eef68126-35a7-4716-a359-5b31ff36b0cf-kube-api-access-5bh5p\") pod \"maas-api-revoke-sub-b828eafc-kt44t\" (UID: \"eef68126-35a7-4716-a359-5b31ff36b0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"
Nov 02 09:58:23.038345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:23.037679    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"
Nov 02 09:58:23.604241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:23.585897    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"]
Nov 02 09:58:23.911434 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:23.911337    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t" event={"ID":"eef68126-35a7-4716-a359-5b31ff36b0cf","Type":"ContainerStarted","Data":"53af93dff8eb75ab7dc6b1ae0f3daa2e258420ef24fb7771e9be23c21acf89ea"}
Nov 02 09:58:24.924938 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:24.924895    4449 generic.go:356] "Generic (PLEG): container finished" podID="eef68126-35a7-4716-a359-5b31ff36b0cf" containerID="cb46dec60164a6575c5ce207626b07921e88eb8091e9f9231a21c5908aec0d83" exitCode=0
Nov 02 09:58:24.925397 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:24.924989    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t" event={"ID":"eef68126-35a7-4716-a359-5b31ff36b0cf","Type":"ContainerDied","Data":"cb46dec60164a6575c5ce207626b07921e88eb8091e9f9231a21c5908aec0d83"}
Nov 02 09:58:25.070136 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:25.070098    4449 scope.go:122] "RemoveContainer" containerID="0c55d35abf7ea2f880c7a62ef88c56893baf885506f64fb2242be446d5c8ca1a"
Nov 02 09:58:25.081551 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:25.081524    4449 scope.go:122] "RemoveContainer" containerID="5137a92025fdf2825257453777f61801e8133822a045fca0be4d8f9ddd667fdc"
Nov 02 09:58:25.093497 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:25.093470    4449 scope.go:122] "RemoveContainer" containerID="5d421fdfd4aefdd55732f9e2a5a28fa1d808731e1c5937419d58225da8744015"
Nov 02 09:58:25.104633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:25.104607    4449 scope.go:122] "RemoveContainer" containerID="ea79ac4fbe4badf97290017a5faf64684995d4546838f450ab50fa66fac7d586"
Nov 02 09:58:25.117477 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:25.117456    4449 scope.go:122] "RemoveContainer" containerID="d62673821730df0adf7ea1e4158b760f9b34fb0e65b3f9a4ce35983bbd183bf2"
Nov 02 09:58:25.128067 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:25.128039    4449 scope.go:122] "RemoveContainer" containerID="59ac81c3a7a970f88fb84d91bb749a225c03f13c11f9780cecd1c3bfb9e7cd94"
Nov 02 09:58:25.139939 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:25.139914    4449 scope.go:122] "RemoveContainer" containerID="94403f81b1ac88f5bac83a2039d02cd4c6df515067a0d9b2d19cf93cd93ec6f9"
Nov 02 09:58:25.171428 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:25.171399    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9" podUID="c2d7f4ec-14a0-450a-9298-c6da1c6fba89" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:58:26.086014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.085989    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"
Nov 02 09:58:26.179614 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.179572    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j" podUID="eec88150-3595-4245-a745-3e5cb6e0455d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:58:26.254989 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.254900    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-openshift-ca\") pod \"eef68126-35a7-4716-a359-5b31ff36b0cf\" (UID: \"eef68126-35a7-4716-a359-5b31ff36b0cf\") "
Nov 02 09:58:26.255152 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.255132    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-xks-ca\") pod \"eef68126-35a7-4716-a359-5b31ff36b0cf\" (UID: \"eef68126-35a7-4716-a359-5b31ff36b0cf\") "
Nov 02 09:58:26.255345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.255327    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef68126-35a7-4716-a359-5b31ff36b0cf-kube-api-access-5bh5p\" (UniqueName: \"kubernetes.io/projected/eef68126-35a7-4716-a359-5b31ff36b0cf-kube-api-access-5bh5p\") pod \"eef68126-35a7-4716-a359-5b31ff36b0cf\" (UID: \"eef68126-35a7-4716-a359-5b31ff36b0cf\") "
Nov 02 09:58:26.255345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.255332    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-openshift-ca" pod "eef68126-35a7-4716-a359-5b31ff36b0cf" (UID: "eef68126-35a7-4716-a359-5b31ff36b0cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:26.255838 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.255815    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:26.257621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.257589    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-xks-ca" pod "eef68126-35a7-4716-a359-5b31ff36b0cf" (UID: "eef68126-35a7-4716-a359-5b31ff36b0cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:58:26.257727 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.257651    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef68126-35a7-4716-a359-5b31ff36b0cf-kube-api-access-5bh5p" pod "eef68126-35a7-4716-a359-5b31ff36b0cf" (UID: "eef68126-35a7-4716-a359-5b31ff36b0cf"). InnerVolumeSpecName "kube-api-access-5bh5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:26.358030 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.358001    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef68126-35a7-4716-a359-5b31ff36b0cf-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:26.358030 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.358028    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bh5p\" (UniqueName: \"kubernetes.io/projected/eef68126-35a7-4716-a359-5b31ff36b0cf-kube-api-access-5bh5p\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:26.944295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.944265    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"
Nov 02 09:58:26.944295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.944272    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t" event={"ID":"eef68126-35a7-4716-a359-5b31ff36b0cf","Type":"ContainerDied","Data":"53af93dff8eb75ab7dc6b1ae0f3daa2e258420ef24fb7771e9be23c21acf89ea"}
Nov 02 09:58:26.944542 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:26.944324    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53af93dff8eb75ab7dc6b1ae0f3daa2e258420ef24fb7771e9be23c21acf89ea"
Nov 02 09:58:28.595070 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:28.595034    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"]
Nov 02 09:58:28.606254 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:28.606193    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"]
Nov 02 09:58:30.073335 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:30.073296    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"]
Nov 02 09:58:30.082376 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:30.082330    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8714c0-5mnnl"]
Nov 02 09:58:31.222003 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:31.221941    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c968779-cd4c-4b68-9aa0-b2f1895a0558" path="/var/lib/kubelet/pods/0c968779-cd4c-4b68-9aa0-b2f1895a0558/volumes"
Nov 02 09:58:33.319692 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.319652    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"]
Nov 02 09:58:33.321415 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.321378    4449 cpu_manager.go:414] "containerMap: removing container" podUID="eef68126-35a7-4716-a359-5b31ff36b0cf" containerName="revoke-keys"
Nov 02 09:58:33.321415 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.321409    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef68126-35a7-4716-a359-5b31ff36b0cf" containerName="revoke-keys"
Nov 02 09:58:33.321628 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.321586    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef68126-35a7-4716-a359-5b31ff36b0cf" containerName="revoke-keys"
Nov 02 09:58:33.327506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.327481    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"]
Nov 02 09:58:33.327648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.327611    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.331605 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.331584    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-629129-openshift-default-dockercfg-mggz7\""
Nov 02 09:58:33.452823 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.452784    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.453037 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.452950    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6a26be-45d7-47e9-a62e-5920e516f037-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.453037 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.453024    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.453173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.453101    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.453259 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.453183    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.453367 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.453342    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6t2h\" (UniqueName: \"kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-kube-api-access-s6t2h\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.453437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.453377    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.453437 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.453401    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.453552 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.453533    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.554440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.554399    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.554677 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.554498    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6a26be-45d7-47e9-a62e-5920e516f037-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.554677 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.554548    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.554677 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.554575    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.554866 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.554711    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.554932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.554859    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6t2h\" (UniqueName: \"kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-kube-api-access-s6t2h\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.554932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.554891    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.554932 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.554926    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.555117 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.555013    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.555769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.555737    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.556027 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.556000    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.556173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.556151    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6a26be-45d7-47e9-a62e-5920e516f037-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.556309 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.556286    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.556417 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.556356    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.558251 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.558193    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.558875 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.558844    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.568819 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.568786    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.568994 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.568964    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6t2h\" (UniqueName: \"kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-kube-api-access-s6t2h\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:33.595634 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.595510    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t" podUID="7fcc4135-f9c4-4c04-96da-26ad2abd502e" containerName="payload-processing" containerID="cri-o://3156aa548a2904e0ade7d04c4d901bc6de93f99bc34a66c365da3a0b07de8673" gracePeriod=25
Nov 02 09:58:33.618186 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.608802    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk" podUID="35d1f11e-bd29-4c15-af90-027c65c690fa" containerName="payload-pre-processing" containerID="cri-o://e3dfe6a5133d381ba3b385b7243890b474b52df8d1898892013a1ea77d4d146d" gracePeriod=25
Nov 02 09:58:33.655735 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:33.650153    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:34.182308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.179506    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"
Nov 02 09:58:34.188113 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.188074    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"]
Nov 02 09:58:34.189761 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.189730    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"
Nov 02 09:58:34.197162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.196210    4449 generic.go:356] "Generic (PLEG): container finished" podID="eec88150-3595-4245-a745-3e5cb6e0455d" containerID="68196c16ea44175392cb383680a792ccef636a0653f024e93fb769a948314916" exitCode=0
Nov 02 09:58:34.197162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.196674    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"
Nov 02 09:58:34.197162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.196769    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j" event={"ID":"eec88150-3595-4245-a745-3e5cb6e0455d","Type":"ContainerDied","Data":"68196c16ea44175392cb383680a792ccef636a0653f024e93fb769a948314916"}
Nov 02 09:58:34.197162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.196822    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j" event={"ID":"eec88150-3595-4245-a745-3e5cb6e0455d","Type":"ContainerDied","Data":"d5beda92820c6e65552fd304d65f4bfa225321a95b25fe5db060859355774e1c"}
Nov 02 09:58:34.197162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.196853    4449 scope.go:122] "RemoveContainer" containerID="68196c16ea44175392cb383680a792ccef636a0653f024e93fb769a948314916"
Nov 02 09:58:34.204517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.203843    4449 generic.go:356] "Generic (PLEG): container finished" podID="c2d7f4ec-14a0-450a-9298-c6da1c6fba89" containerID="4067e0133864cc21ce45e35d3dd3338699b9d139773c83f2fd32969045d00a63" exitCode=0
Nov 02 09:58:34.204517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.203876    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9" event={"ID":"c2d7f4ec-14a0-450a-9298-c6da1c6fba89","Type":"ContainerDied","Data":"4067e0133864cc21ce45e35d3dd3338699b9d139773c83f2fd32969045d00a63"}
Nov 02 09:58:34.204517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.204366    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"
Nov 02 09:58:34.224877 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.224851    4449 scope.go:122] "RemoveContainer" containerID="68196c16ea44175392cb383680a792ccef636a0653f024e93fb769a948314916"
Nov 02 09:58:34.225201 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:34.225178    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68196c16ea44175392cb383680a792ccef636a0653f024e93fb769a948314916\": container with ID starting with 68196c16ea44175392cb383680a792ccef636a0653f024e93fb769a948314916 not found: ID does not exist" containerID="68196c16ea44175392cb383680a792ccef636a0653f024e93fb769a948314916"
Nov 02 09:58:34.225367 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.225253    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68196c16ea44175392cb383680a792ccef636a0653f024e93fb769a948314916"} err="failed to get container status \"68196c16ea44175392cb383680a792ccef636a0653f024e93fb769a948314916\": rpc error: code = NotFound desc = could not find container \"68196c16ea44175392cb383680a792ccef636a0653f024e93fb769a948314916\": container with ID starting with 68196c16ea44175392cb383680a792ccef636a0653f024e93fb769a948314916 not found: ID does not exist"
Nov 02 09:58:34.225367 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.225281    4449 scope.go:122] "RemoveContainer" containerID="4067e0133864cc21ce45e35d3dd3338699b9d139773c83f2fd32969045d00a63"
Nov 02 09:58:34.225479 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.225357    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:58:34.225479 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.225425    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:58:34.225479 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.225456    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:58:34.276352 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.276324    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"]
Nov 02 09:58:34.277137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.277116    4449 cpu_manager.go:414] "containerMap: removing container" podUID="eec88150-3595-4245-a745-3e5cb6e0455d" containerName="payload-pre-processing"
Nov 02 09:58:34.277137 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.277137    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec88150-3595-4245-a745-3e5cb6e0455d" containerName="payload-pre-processing"
Nov 02 09:58:34.277349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.277147    4449 cpu_manager.go:414] "containerMap: removing container" podUID="c2d7f4ec-14a0-450a-9298-c6da1c6fba89" containerName="payload-processing"
Nov 02 09:58:34.277349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.277151    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d7f4ec-14a0-450a-9298-c6da1c6fba89" containerName="payload-processing"
Nov 02 09:58:34.277349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.277275    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec88150-3595-4245-a745-3e5cb6e0455d" containerName="payload-pre-processing"
Nov 02 09:58:34.277349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.277284    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d7f4ec-14a0-450a-9298-c6da1c6fba89" containerName="payload-processing"
Nov 02 09:58:34.286584 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.284312    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"
Nov 02 09:58:34.291843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.291764    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:58:34.292490 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.291968    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:58:34.292490 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.292033    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:58:34.292490 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.292129    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:58:34.292877 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.292859    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"]
Nov 02 09:58:34.295938 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.295772    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec88150-3595-4245-a745-3e5cb6e0455d-kube-api-access-4dcwl\" (UniqueName: \"kubernetes.io/projected/eec88150-3595-4245-a745-3e5cb6e0455d-kube-api-access-4dcwl\") pod \"eec88150-3595-4245-a745-3e5cb6e0455d\" (UID: \"eec88150-3595-4245-a745-3e5cb6e0455d\") "
Nov 02 09:58:34.295938 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.295892    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec88150-3595-4245-a745-3e5cb6e0455d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec88150-3595-4245-a745-3e5cb6e0455d-plugins-config-volume\") pod \"eec88150-3595-4245-a745-3e5cb6e0455d\" (UID: \"eec88150-3595-4245-a745-3e5cb6e0455d\") "
Nov 02 09:58:34.296150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.295957    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-plugins-config-volume\") pod \"c2d7f4ec-14a0-450a-9298-c6da1c6fba89\" (UID: \"c2d7f4ec-14a0-450a-9298-c6da1c6fba89\") "
Nov 02 09:58:34.296150 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.295997    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-kube-api-access-79m6g\" (UniqueName: \"kubernetes.io/projected/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-kube-api-access-79m6g\") pod \"c2d7f4ec-14a0-450a-9298-c6da1c6fba89\" (UID: \"c2d7f4ec-14a0-450a-9298-c6da1c6fba89\") "
Nov 02 09:58:34.299541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.299151    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-plugins-config-volume" pod "c2d7f4ec-14a0-450a-9298-c6da1c6fba89" (UID: "c2d7f4ec-14a0-450a-9298-c6da1c6fba89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:34.300651 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.300575    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec88150-3595-4245-a745-3e5cb6e0455d-plugins-config-volume" pod "eec88150-3595-4245-a745-3e5cb6e0455d" (UID: "eec88150-3595-4245-a745-3e5cb6e0455d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:34.302183 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.302143    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec88150-3595-4245-a745-3e5cb6e0455d-kube-api-access-4dcwl" pod "eec88150-3595-4245-a745-3e5cb6e0455d" (UID: "eec88150-3595-4245-a745-3e5cb6e0455d"). InnerVolumeSpecName "kube-api-access-4dcwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:34.305401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.305335    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-kube-api-access-79m6g" pod "c2d7f4ec-14a0-450a-9298-c6da1c6fba89" (UID: "c2d7f4ec-14a0-450a-9298-c6da1c6fba89"). InnerVolumeSpecName "kube-api-access-79m6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:34.398960 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.398835    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d429e290-m8jz7\" (UID: \"776a2c05-d364-450b-87ed-fbcf7554a5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"
Nov 02 09:58:34.399446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.398962    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8zgn\" (UniqueName: \"kubernetes.io/projected/776a2c05-d364-450b-87ed-fbcf7554a5c2-kube-api-access-z8zgn\") pod \"maas-api-revoke-sub-d429e290-m8jz7\" (UID: \"776a2c05-d364-450b-87ed-fbcf7554a5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"
Nov 02 09:58:34.399446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.399063    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d429e290-m8jz7\" (UID: \"776a2c05-d364-450b-87ed-fbcf7554a5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"
Nov 02 09:58:34.399446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.399406    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dcwl\" (UniqueName: \"kubernetes.io/projected/eec88150-3595-4245-a745-3e5cb6e0455d-kube-api-access-4dcwl\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:34.399446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.399431    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec88150-3595-4245-a745-3e5cb6e0455d-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:34.399446 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.399446    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:34.399643 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.399459    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79m6g\" (UniqueName: \"kubernetes.io/projected/c2d7f4ec-14a0-450a-9298-c6da1c6fba89-kube-api-access-79m6g\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:34.501538 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.501505    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d429e290-m8jz7\" (UID: \"776a2c05-d364-450b-87ed-fbcf7554a5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"
Nov 02 09:58:34.501743 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.501560    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8zgn\" (UniqueName: \"kubernetes.io/projected/776a2c05-d364-450b-87ed-fbcf7554a5c2-kube-api-access-z8zgn\") pod \"maas-api-revoke-sub-d429e290-m8jz7\" (UID: \"776a2c05-d364-450b-87ed-fbcf7554a5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"
Nov 02 09:58:34.501743 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.501615    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d429e290-m8jz7\" (UID: \"776a2c05-d364-450b-87ed-fbcf7554a5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"
Nov 02 09:58:34.502462 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.502424    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d429e290-m8jz7\" (UID: \"776a2c05-d364-450b-87ed-fbcf7554a5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"
Nov 02 09:58:34.504679 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.504651    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d429e290-m8jz7\" (UID: \"776a2c05-d364-450b-87ed-fbcf7554a5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"
Nov 02 09:58:34.514682 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.514647    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8zgn\" (UniqueName: \"kubernetes.io/projected/776a2c05-d364-450b-87ed-fbcf7554a5c2-kube-api-access-z8zgn\") pod \"maas-api-revoke-sub-d429e290-m8jz7\" (UID: \"776a2c05-d364-450b-87ed-fbcf7554a5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"
Nov 02 09:58:34.532714 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.532679    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"]
Nov 02 09:58:34.540753 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.540716    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2e80fb-77fd4bc6bc-ncz2j"]
Nov 02 09:58:34.550381 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.550293    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"]
Nov 02 09:58:34.553841 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.553805    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2e80fb-7f8cb49b89-d2pv9"]
Nov 02 09:58:34.615104 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.615074    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"
Nov 02 09:58:34.922613 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:58:34.922580    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776a2c05_d364_450b_87ed_fbcf7554a5c2.slice/crio-628db451dad5441029479e160b4e7ea96f6b604826cf2fb20d08faa0fa9a706f WatchSource:0}: Error finding container 628db451dad5441029479e160b4e7ea96f6b604826cf2fb20d08faa0fa9a706f: Status 404 returned error can't find the container with id 628db451dad5441029479e160b4e7ea96f6b604826cf2fb20d08faa0fa9a706f
Nov 02 09:58:34.922915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:34.922893    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"]
Nov 02 09:58:35.207197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:35.207158    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d7f4ec-14a0-450a-9298-c6da1c6fba89" path="/var/lib/kubelet/pods/c2d7f4ec-14a0-450a-9298-c6da1c6fba89/volumes"
Nov 02 09:58:35.207740 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:35.207722    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec88150-3595-4245-a745-3e5cb6e0455d" path="/var/lib/kubelet/pods/eec88150-3595-4245-a745-3e5cb6e0455d/volumes"
Nov 02 09:58:35.214290 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:35.214259    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7" event={"ID":"776a2c05-d364-450b-87ed-fbcf7554a5c2","Type":"ContainerStarted","Data":"628db451dad5441029479e160b4e7ea96f6b604826cf2fb20d08faa0fa9a706f"}
Nov 02 09:58:35.216083 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:35.216053    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s" event={"ID":"4b6a26be-45d7-47e9-a62e-5920e516f037","Type":"ContainerStarted","Data":"aa8855b994a1ed196f1597e5ff7285733bf215c8a77e395ac384ba76fccdaa9c"}
Nov 02 09:58:35.216271 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:35.216102    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s" event={"ID":"4b6a26be-45d7-47e9-a62e-5920e516f037","Type":"ContainerStarted","Data":"b9ad36d4ec3a38282dbd078ad0520b6e37d2d7c3ade735431e44a50033c4f8d6"}
Nov 02 09:58:35.240409 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:35.240285    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s" podStartSLOduration=2.240268552 podStartE2EDuration="2.240268552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:58:35.238564365 +0000 UTC m=+1392.176851165" watchObservedRunningTime="2026-11-02 09:58:35.240268552 +0000 UTC m=+1392.178555343"
Nov 02 09:58:35.674289 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:35.670950    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:35.676307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:35.676276    4449 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 02 09:58:35.676472 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:35.676337    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s" podUID="4b6a26be-45d7-47e9-a62e-5920e516f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 02 09:58:36.025979 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.025890    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"]
Nov 02 09:58:36.030828 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.030795    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87af7f2f-wcj92"]
Nov 02 09:58:36.228081 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.228045    4449 generic.go:356] "Generic (PLEG): container finished" podID="776a2c05-d364-450b-87ed-fbcf7554a5c2" containerID="a2ad9b0a1d0c1f2d06ea9d26bd730a3adc6e3fcdf16326b93a902c838d599b10" exitCode=0
Nov 02 09:58:36.228315 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.228075    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7" event={"ID":"776a2c05-d364-450b-87ed-fbcf7554a5c2","Type":"ContainerDied","Data":"a2ad9b0a1d0c1f2d06ea9d26bd730a3adc6e3fcdf16326b93a902c838d599b10"}
Nov 02 09:58:36.548487 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:36.548442    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:58:36.685060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.685013    4449 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 02 09:58:36.685622 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.685086    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s" podUID="4b6a26be-45d7-47e9-a62e-5920e516f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 02 09:58:36.692821 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.692792    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"]
Nov 02 09:58:36.693049 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.693024    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerName="istio-proxy" containerID="cri-o://45373cd1ec8060d63bf6c6f42240f34ed8de3daab98beb28700607f02cb0ff4f" gracePeriod=30
Nov 02 09:58:36.699664 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.699636    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"]
Nov 02 09:58:36.699887 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.699854    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerName="istio-proxy" containerID="cri-o://62db54214e8204fa57c4e62a8114ba135017de1f898ba9d40061ec25d44fb52d" gracePeriod=30
Nov 02 09:58:36.704308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.704284    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"]
Nov 02 09:58:36.704632 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.704493    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc" podUID="bdbc7509-08cd-45ab-ad9d-32527a7c48af" containerName="istio-proxy" containerID="cri-o://547acb298827aa6664e7cf78af29280b222bc99176569af6ea09ff61e8f178f6" gracePeriod=30
Nov 02 09:58:36.709229 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.709190    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"]
Nov 02 09:58:36.709502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:36.709479    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerName="istio-proxy" containerID="cri-o://ce4537aa0227c6323fb9eb3be763dfcca4cd680f5785062d98390f41ccfaf626" gracePeriod=30
Nov 02 09:58:37.208609 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:37.208575    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d5d07c-71f9-455b-934a-05d4a2e7303c" path="/var/lib/kubelet/pods/71d5d07c-71f9-455b-934a-05d4a2e7303c/volumes"
Nov 02 09:58:37.576292 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:37.569620    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"
Nov 02 09:58:37.673246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:37.663828    4449 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 02 09:58:37.673246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:37.663889    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s" podUID="4b6a26be-45d7-47e9-a62e-5920e516f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 02 09:58:37.716767 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:37.716738    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-openshift-ca\") pod \"776a2c05-d364-450b-87ed-fbcf7554a5c2\" (UID: \"776a2c05-d364-450b-87ed-fbcf7554a5c2\") "
Nov 02 09:58:37.717153 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:37.716804    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776a2c05-d364-450b-87ed-fbcf7554a5c2-kube-api-access-z8zgn\" (UniqueName: \"kubernetes.io/projected/776a2c05-d364-450b-87ed-fbcf7554a5c2-kube-api-access-z8zgn\") pod \"776a2c05-d364-450b-87ed-fbcf7554a5c2\" (UID: \"776a2c05-d364-450b-87ed-fbcf7554a5c2\") "
Nov 02 09:58:37.717153 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:37.716904    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-xks-ca\") pod \"776a2c05-d364-450b-87ed-fbcf7554a5c2\" (UID: \"776a2c05-d364-450b-87ed-fbcf7554a5c2\") "
Nov 02 09:58:37.717280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:37.717153    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-openshift-ca" pod "776a2c05-d364-450b-87ed-fbcf7554a5c2" (UID: "776a2c05-d364-450b-87ed-fbcf7554a5c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:37.717588 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:37.717567    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:37.719116 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:37.719091    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-xks-ca" pod "776a2c05-d364-450b-87ed-fbcf7554a5c2" (UID: "776a2c05-d364-450b-87ed-fbcf7554a5c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:58:37.719172 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:37.719134    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776a2c05-d364-450b-87ed-fbcf7554a5c2-kube-api-access-z8zgn" pod "776a2c05-d364-450b-87ed-fbcf7554a5c2" (UID: "776a2c05-d364-450b-87ed-fbcf7554a5c2"). InnerVolumeSpecName "kube-api-access-z8zgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:37.818760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:37.818727    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8zgn\" (UniqueName: \"kubernetes.io/projected/776a2c05-d364-450b-87ed-fbcf7554a5c2-kube-api-access-z8zgn\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:37.818760 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:37.818756    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/776a2c05-d364-450b-87ed-fbcf7554a5c2-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:38.245344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:38.245250    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"
Nov 02 09:58:38.245344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:38.245266    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7" event={"ID":"776a2c05-d364-450b-87ed-fbcf7554a5c2","Type":"ContainerDied","Data":"628db451dad5441029479e160b4e7ea96f6b604826cf2fb20d08faa0fa9a706f"}
Nov 02 09:58:38.245344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:38.245325    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="628db451dad5441029479e160b4e7ea96f6b604826cf2fb20d08faa0fa9a706f"
Nov 02 09:58:38.674781 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:38.674730    4449 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 02 09:58:38.674979 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:38.674810    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s" podUID="4b6a26be-45d7-47e9-a62e-5920e516f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 02 09:58:38.820940 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:38.820888    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk" podUID="35d1f11e-bd29-4c15-af90-027c65c690fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:58:39.671245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:39.665830    4449 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 02 09:58:39.671245 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:39.665906    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s" podUID="4b6a26be-45d7-47e9-a62e-5920e516f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 02 09:58:39.827124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:39.827090    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t" podUID="7fcc4135-f9c4-4c04-96da-26ad2abd502e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:58:40.035609 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:40.035566    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"]
Nov 02 09:58:40.040765 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:40.040736    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dab2b34d-v485c"]
Nov 02 09:58:40.664494 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:40.664453    4449 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 02 09:58:40.664671 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:40.664523    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s" podUID="4b6a26be-45d7-47e9-a62e-5920e516f037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 02 09:58:41.212434 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:41.212396    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6552c1-e93c-406e-8749-5796468c5aba" path="/var/lib/kubelet/pods/ed6552c1-e93c-406e-8749-5796468c5aba/volumes"
Nov 02 09:58:41.668691 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:41.668660    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:41.669153 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:41.669139    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:41.669915 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:41.669894    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 09:58:42.050283 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.047689    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:58:42.144976 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.144948    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:58:42.151704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.150718    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-data\") pod \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") "
Nov 02 09:58:42.151704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.150784    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-token\" (UniqueName: \"kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-token\") pod \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") "
Nov 02 09:58:42.151704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.150822    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-podinfo\") pod \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") "
Nov 02 09:58:42.151704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.150882    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-kube-api-access-f9csh\" (UniqueName: \"kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-kube-api-access-f9csh\") pod \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") "
Nov 02 09:58:42.151704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.151001    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-certs\") pod \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") "
Nov 02 09:58:42.151704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.151085    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istiod-ca-cert\") pod \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") "
Nov 02 09:58:42.151704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.151183    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-credential-socket\") pod \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") "
Nov 02 09:58:42.151704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.151244    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-envoy\") pod \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") "
Nov 02 09:58:42.151704 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.151301    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-socket\") pod \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\" (UID: \"2ab2b6c8-fd73-43f8-811c-d31f347557bf\") "
Nov 02 09:58:42.153592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.153282    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-socket" pod "2ab2b6c8-fd73-43f8-811c-d31f347557bf" (UID: "2ab2b6c8-fd73-43f8-811c-d31f347557bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.155190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.154147    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-data" pod "2ab2b6c8-fd73-43f8-811c-d31f347557bf" (UID: "2ab2b6c8-fd73-43f8-811c-d31f347557bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.155190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.154193    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-credential-socket" pod "2ab2b6c8-fd73-43f8-811c-d31f347557bf" (UID: "2ab2b6c8-fd73-43f8-811c-d31f347557bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.155190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.154345    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-certs" pod "2ab2b6c8-fd73-43f8-811c-d31f347557bf" (UID: "2ab2b6c8-fd73-43f8-811c-d31f347557bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.155190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.154536    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istiod-ca-cert" pod "2ab2b6c8-fd73-43f8-811c-d31f347557bf" (UID: "2ab2b6c8-fd73-43f8-811c-d31f347557bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:42.155619 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.155601    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:58:42.157845 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.157772    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-podinfo" pod "2ab2b6c8-fd73-43f8-811c-d31f347557bf" (UID: "2ab2b6c8-fd73-43f8-811c-d31f347557bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:42.158294 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.158267    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-kube-api-access-f9csh" pod "2ab2b6c8-fd73-43f8-811c-d31f347557bf" (UID: "2ab2b6c8-fd73-43f8-811c-d31f347557bf"). InnerVolumeSpecName "kube-api-access-f9csh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.159279 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.159242    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-envoy" pod "2ab2b6c8-fd73-43f8-811c-d31f347557bf" (UID: "2ab2b6c8-fd73-43f8-811c-d31f347557bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.161093 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.161055    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-token" pod "2ab2b6c8-fd73-43f8-811c-d31f347557bf" (UID: "2ab2b6c8-fd73-43f8-811c-d31f347557bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.168735 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.168702    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"]
Nov 02 09:58:42.169000 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.168948    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerName="istio-proxy" containerID="cri-o://eee7d9cba4cfe5574e4893f6da68f0097b6872d95745e4271778dc19318463cc" gracePeriod=30
Nov 02 09:58:42.170924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.170896    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:58:42.179419 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.178907    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"]
Nov 02 09:58:42.179419 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.179103    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerName="istio-proxy" containerID="cri-o://edb6b950f6cf882054c331f8062ef962daffd73a3554403e7b35df3addf5ff75" gracePeriod=30
Nov 02 09:58:42.214871 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.214821    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"]
Nov 02 09:58:42.215385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.215082    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" podUID="20132bdb-f0fa-4408-9a26-5b34c1510038" containerName="istio-proxy" containerID="cri-o://e6f02ebcdcac78aceffa25695b576b3034ffcb0aa94746384ed7bd0bcd6a3ac5" gracePeriod=30
Nov 02 09:58:42.253566 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.253483    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-socket\") pod \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") "
Nov 02 09:58:42.258358 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.253990    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-socket" pod "a0a81758-9db8-4221-9ab7-7a20d2400a2d" (UID: "a0a81758-9db8-4221-9ab7-7a20d2400a2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.258939 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.258882    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istiod-ca-cert" pod "a1bccd9c-ca7e-4f26-81da-3d5d77be08df" (UID: "a1bccd9c-ca7e-4f26-81da-3d5d77be08df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:42.258939 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.258549    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istiod-ca-cert\") pod \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") "
Nov 02 09:58:42.265162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.259050    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-token\" (UniqueName: \"kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-token\") pod \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") "
Nov 02 09:58:42.265162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.259116    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-data\") pod \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") "
Nov 02 09:58:42.265162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.259146    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-certs\") pod \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") "
Nov 02 09:58:42.265162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.259171    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-socket\") pod \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") "
Nov 02 09:58:42.265162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.259200    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-podinfo\") pod \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") "
Nov 02 09:58:42.265162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.260025    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-certs" pod "a0a81758-9db8-4221-9ab7-7a20d2400a2d" (UID: "a0a81758-9db8-4221-9ab7-7a20d2400a2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.265162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.260182    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-socket" pod "a1bccd9c-ca7e-4f26-81da-3d5d77be08df" (UID: "a1bccd9c-ca7e-4f26-81da-3d5d77be08df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.265162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.260301    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-data" pod "a0a81758-9db8-4221-9ab7-7a20d2400a2d" (UID: "a0a81758-9db8-4221-9ab7-7a20d2400a2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.265162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.260601    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-token\" (UniqueName: \"kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-token\") pod \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") "
Nov 02 09:58:42.265162 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.263086    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-token" pod "bdbc7509-08cd-45ab-ad9d-32527a7c48af" (UID: "bdbc7509-08cd-45ab-ad9d-32527a7c48af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.266039 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.265321    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-token" pod "a1bccd9c-ca7e-4f26-81da-3d5d77be08df" (UID: "a1bccd9c-ca7e-4f26-81da-3d5d77be08df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.267657 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.267626    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-podinfo" pod "a0a81758-9db8-4221-9ab7-7a20d2400a2d" (UID: "a0a81758-9db8-4221-9ab7-7a20d2400a2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:42.273128 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.273077    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-data" pod "a1bccd9c-ca7e-4f26-81da-3d5d77be08df" (UID: "a1bccd9c-ca7e-4f26-81da-3d5d77be08df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.278345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278316    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-data\") pod \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") "
Nov 02 09:58:42.278491 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278382    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-envoy\") pod \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") "
Nov 02 09:58:42.278491 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278423    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-podinfo\") pod \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") "
Nov 02 09:58:42.278491 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278479    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istiod-ca-cert\") pod \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") "
Nov 02 09:58:42.278727 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278507    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-podinfo\") pod \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") "
Nov 02 09:58:42.278727 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278548    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-data\") pod \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") "
Nov 02 09:58:42.278727 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278580    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-kube-api-access-bkmc8\" (UniqueName: \"kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-kube-api-access-bkmc8\") pod \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") "
Nov 02 09:58:42.278727 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278614    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-certs\") pod \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") "
Nov 02 09:58:42.278727 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278655    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istiod-ca-cert\") pod \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") "
Nov 02 09:58:42.278727 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278689    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-certs\") pod \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") "
Nov 02 09:58:42.279059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278733    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-kube-api-access-dn84l\" (UniqueName: \"kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-kube-api-access-dn84l\") pod \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") "
Nov 02 09:58:42.279059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278767    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-socket\") pod \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") "
Nov 02 09:58:42.279059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278799    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-credential-socket\") pod \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") "
Nov 02 09:58:42.279059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278851    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-envoy\") pod \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") "
Nov 02 09:58:42.279059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278928    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-credential-socket\") pod \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") "
Nov 02 09:58:42.279059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.278976    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-token\" (UniqueName: \"kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-token\") pod \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") "
Nov 02 09:58:42.279059 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279006    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-credential-socket\") pod \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\" (UID: \"bdbc7509-08cd-45ab-ad9d-32527a7c48af\") "
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279064    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-envoy\") pod \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\" (UID: \"a0a81758-9db8-4221-9ab7-7a20d2400a2d\") "
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279113    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-kube-api-access-ctvg8\" (UniqueName: \"kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-kube-api-access-ctvg8\") pod \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\" (UID: \"a1bccd9c-ca7e-4f26-81da-3d5d77be08df\") "
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279802    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-socket" pod "bdbc7509-08cd-45ab-ad9d-32527a7c48af" (UID: "bdbc7509-08cd-45ab-ad9d-32527a7c48af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279822    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279848    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279863    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279877    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279890    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279902    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279915    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279928    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279943    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab2b6c8-fd73-43f8-811c-d31f347557bf-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279956    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279970    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9csh\" (UniqueName: \"kubernetes.io/projected/2ab2b6c8-fd73-43f8-811c-d31f347557bf-kube-api-access-f9csh\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279984    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279987    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-data" pod "bdbc7509-08cd-45ab-ad9d-32527a7c48af" (UID: "bdbc7509-08cd-45ab-ad9d-32527a7c48af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.279998    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.280012    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.280026    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.280041    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.280585 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.280056    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab2b6c8-fd73-43f8-811c-d31f347557bf-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.289543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.285497    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istiod-ca-cert" pod "bdbc7509-08cd-45ab-ad9d-32527a7c48af" (UID: "bdbc7509-08cd-45ab-ad9d-32527a7c48af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:42.303320 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.298525    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-certs" pod "bdbc7509-08cd-45ab-ad9d-32527a7c48af" (UID: "bdbc7509-08cd-45ab-ad9d-32527a7c48af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.303320 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.298871    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istiod-ca-cert" pod "a0a81758-9db8-4221-9ab7-7a20d2400a2d" (UID: "a0a81758-9db8-4221-9ab7-7a20d2400a2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:42.303320 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.302914    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-certs" pod "a1bccd9c-ca7e-4f26-81da-3d5d77be08df" (UID: "a1bccd9c-ca7e-4f26-81da-3d5d77be08df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.306924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.306169    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-credential-socket" pod "a0a81758-9db8-4221-9ab7-7a20d2400a2d" (UID: "a0a81758-9db8-4221-9ab7-7a20d2400a2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.306924 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.306387    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-credential-socket" pod "bdbc7509-08cd-45ab-ad9d-32527a7c48af" (UID: "bdbc7509-08cd-45ab-ad9d-32527a7c48af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.320026 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.319993    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-credential-socket" pod "a1bccd9c-ca7e-4f26-81da-3d5d77be08df" (UID: "a1bccd9c-ca7e-4f26-81da-3d5d77be08df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.333983 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.333926    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-kube-api-access-dn84l" pod "bdbc7509-08cd-45ab-ad9d-32527a7c48af" (UID: "bdbc7509-08cd-45ab-ad9d-32527a7c48af"). InnerVolumeSpecName "kube-api-access-dn84l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.335243 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.334299    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-podinfo" pod "bdbc7509-08cd-45ab-ad9d-32527a7c48af" (UID: "bdbc7509-08cd-45ab-ad9d-32527a7c48af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:42.337576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.337544    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-kube-api-access-bkmc8" pod "a0a81758-9db8-4221-9ab7-7a20d2400a2d" (UID: "a0a81758-9db8-4221-9ab7-7a20d2400a2d"). InnerVolumeSpecName "kube-api-access-bkmc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.337710 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.337627    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-kube-api-access-ctvg8" pod "a1bccd9c-ca7e-4f26-81da-3d5d77be08df" (UID: "a1bccd9c-ca7e-4f26-81da-3d5d77be08df"). InnerVolumeSpecName "kube-api-access-ctvg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.340951 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.340825    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-envoy" pod "bdbc7509-08cd-45ab-ad9d-32527a7c48af" (UID: "bdbc7509-08cd-45ab-ad9d-32527a7c48af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.364909 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.364241    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-token" pod "a0a81758-9db8-4221-9ab7-7a20d2400a2d" (UID: "a0a81758-9db8-4221-9ab7-7a20d2400a2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.364909 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.364347    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-podinfo" pod "a1bccd9c-ca7e-4f26-81da-3d5d77be08df" (UID: "a1bccd9c-ca7e-4f26-81da-3d5d77be08df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:42.365782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.365740    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-envoy" pod "a0a81758-9db8-4221-9ab7-7a20d2400a2d" (UID: "a0a81758-9db8-4221-9ab7-7a20d2400a2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.381541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381441    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381476    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381493    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381508    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381521    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381541 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381537    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkmc8\" (UniqueName: \"kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-kube-api-access-bkmc8\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381554    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381568    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381582    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381596    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn84l\" (UniqueName: \"kubernetes.io/projected/bdbc7509-08cd-45ab-ad9d-32527a7c48af-kube-api-access-dn84l\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381611    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381625    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381638    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381654    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381669    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381684    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbc7509-08cd-45ab-ad9d-32527a7c48af-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381698    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a81758-9db8-4221-9ab7-7a20d2400a2d-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.381914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.381712    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctvg8\" (UniqueName: \"kubernetes.io/projected/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-kube-api-access-ctvg8\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.382858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.382813    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-envoy" pod "a1bccd9c-ca7e-4f26-81da-3d5d77be08df" (UID: "a1bccd9c-ca7e-4f26-81da-3d5d77be08df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.389080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.389047    4449 generic.go:356] "Generic (PLEG): container finished" podID="bdbc7509-08cd-45ab-ad9d-32527a7c48af" containerID="547acb298827aa6664e7cf78af29280b222bc99176569af6ea09ff61e8f178f6" exitCode=0
Nov 02 09:58:42.389264 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.389158    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc" event={"ID":"bdbc7509-08cd-45ab-ad9d-32527a7c48af","Type":"ContainerDied","Data":"547acb298827aa6664e7cf78af29280b222bc99176569af6ea09ff61e8f178f6"}
Nov 02 09:58:42.389264 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.389208    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc" event={"ID":"bdbc7509-08cd-45ab-ad9d-32527a7c48af","Type":"ContainerDied","Data":"95cb4e8e88e85a72d6752a010374c8de1b361bb8d1c30a2f98efb24c11bdfb0f"}
Nov 02 09:58:42.389264 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.389254    4449 scope.go:122] "RemoveContainer" containerID="547acb298827aa6664e7cf78af29280b222bc99176569af6ea09ff61e8f178f6"
Nov 02 09:58:42.389467 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.389423    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"
Nov 02 09:58:42.393615 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.393586    4449 generic.go:356] "Generic (PLEG): container finished" podID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerID="ce4537aa0227c6323fb9eb3be763dfcca4cd680f5785062d98390f41ccfaf626" exitCode=0
Nov 02 09:58:42.398816 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.393650    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" event={"ID":"a1bccd9c-ca7e-4f26-81da-3d5d77be08df","Type":"ContainerDied","Data":"ce4537aa0227c6323fb9eb3be763dfcca4cd680f5785062d98390f41ccfaf626"}
Nov 02 09:58:42.398816 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.393881    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7" event={"ID":"a1bccd9c-ca7e-4f26-81da-3d5d77be08df","Type":"ContainerDied","Data":"3c3a524ccb280649dfdd1d3ed951ca08a5b32c70350002391bde8a06dfbe5329"}
Nov 02 09:58:42.398816 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.393668    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"
Nov 02 09:58:42.416648 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.416617    4449 generic.go:356] "Generic (PLEG): container finished" podID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerID="62db54214e8204fa57c4e62a8114ba135017de1f898ba9d40061ec25d44fb52d" exitCode=0
Nov 02 09:58:42.416764 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.416724    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" event={"ID":"2ab2b6c8-fd73-43f8-811c-d31f347557bf","Type":"ContainerDied","Data":"62db54214e8204fa57c4e62a8114ba135017de1f898ba9d40061ec25d44fb52d"}
Nov 02 09:58:42.416840 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.416773    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx" event={"ID":"2ab2b6c8-fd73-43f8-811c-d31f347557bf","Type":"ContainerDied","Data":"99bcfa9b438a9004621bb1768f71d12874879885709d47c43d18ba58cb8a3bc9"}
Nov 02 09:58:42.416904 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.416879    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"
Nov 02 09:58:42.418960 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.418854    4449 generic.go:356] "Generic (PLEG): container finished" podID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerID="45373cd1ec8060d63bf6c6f42240f34ed8de3daab98beb28700607f02cb0ff4f" exitCode=0
Nov 02 09:58:42.419358 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.419329    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" event={"ID":"a0a81758-9db8-4221-9ab7-7a20d2400a2d","Type":"ContainerDied","Data":"45373cd1ec8060d63bf6c6f42240f34ed8de3daab98beb28700607f02cb0ff4f"}
Nov 02 09:58:42.419516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.419499    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj" event={"ID":"a0a81758-9db8-4221-9ab7-7a20d2400a2d","Type":"ContainerDied","Data":"36aaa463c3cd50868d0bcb8feffe4a0da61d6126ffd20544f95eb3c41c7b479d"}
Nov 02 09:58:42.419651 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.419369    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"
Nov 02 09:58:42.420020 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.419361    4449 scope.go:122] "RemoveContainer" containerID="547acb298827aa6664e7cf78af29280b222bc99176569af6ea09ff61e8f178f6"
Nov 02 09:58:42.420518 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:42.420496    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547acb298827aa6664e7cf78af29280b222bc99176569af6ea09ff61e8f178f6\": container with ID starting with 547acb298827aa6664e7cf78af29280b222bc99176569af6ea09ff61e8f178f6 not found: ID does not exist" containerID="547acb298827aa6664e7cf78af29280b222bc99176569af6ea09ff61e8f178f6"
Nov 02 09:58:42.420671 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.420648    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547acb298827aa6664e7cf78af29280b222bc99176569af6ea09ff61e8f178f6"} err="failed to get container status \"547acb298827aa6664e7cf78af29280b222bc99176569af6ea09ff61e8f178f6\": rpc error: code = NotFound desc = could not find container \"547acb298827aa6664e7cf78af29280b222bc99176569af6ea09ff61e8f178f6\": container with ID starting with 547acb298827aa6664e7cf78af29280b222bc99176569af6ea09ff61e8f178f6 not found: ID does not exist"
Nov 02 09:58:42.420774 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.420761    4449 scope.go:122] "RemoveContainer" containerID="ce4537aa0227c6323fb9eb3be763dfcca4cd680f5785062d98390f41ccfaf626"
Nov 02 09:58:42.434048 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.434022    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"]
Nov 02 09:58:42.436374 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.436353    4449 scope.go:122] "RemoveContainer" containerID="ce4537aa0227c6323fb9eb3be763dfcca4cd680f5785062d98390f41ccfaf626"
Nov 02 09:58:42.436720 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:42.436699    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4537aa0227c6323fb9eb3be763dfcca4cd680f5785062d98390f41ccfaf626\": container with ID starting with ce4537aa0227c6323fb9eb3be763dfcca4cd680f5785062d98390f41ccfaf626 not found: ID does not exist" containerID="ce4537aa0227c6323fb9eb3be763dfcca4cd680f5785062d98390f41ccfaf626"
Nov 02 09:58:42.436825 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.436728    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4537aa0227c6323fb9eb3be763dfcca4cd680f5785062d98390f41ccfaf626"} err="failed to get container status \"ce4537aa0227c6323fb9eb3be763dfcca4cd680f5785062d98390f41ccfaf626\": rpc error: code = NotFound desc = could not find container \"ce4537aa0227c6323fb9eb3be763dfcca4cd680f5785062d98390f41ccfaf626\": container with ID starting with ce4537aa0227c6323fb9eb3be763dfcca4cd680f5785062d98390f41ccfaf626 not found: ID does not exist"
Nov 02 09:58:42.436825 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.436750    4449 scope.go:122] "RemoveContainer" containerID="62db54214e8204fa57c4e62a8114ba135017de1f898ba9d40061ec25d44fb52d"
Nov 02 09:58:42.440472 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.440444    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-lmdfc"]
Nov 02 09:58:42.450421 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.450312    4449 scope.go:122] "RemoveContainer" containerID="f458b4a3b282515d2613beb5e7dc1d805f82b60b967d39b0ab0b937410cdeb00"
Nov 02 09:58:42.453843 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.453819    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"]
Nov 02 09:58:42.462209 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.462181    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-5lrv7"]
Nov 02 09:58:42.462705 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.462687    4449 scope.go:122] "RemoveContainer" containerID="62db54214e8204fa57c4e62a8114ba135017de1f898ba9d40061ec25d44fb52d"
Nov 02 09:58:42.462959 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:42.462940    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62db54214e8204fa57c4e62a8114ba135017de1f898ba9d40061ec25d44fb52d\": container with ID starting with 62db54214e8204fa57c4e62a8114ba135017de1f898ba9d40061ec25d44fb52d not found: ID does not exist" containerID="62db54214e8204fa57c4e62a8114ba135017de1f898ba9d40061ec25d44fb52d"
Nov 02 09:58:42.463014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.462967    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62db54214e8204fa57c4e62a8114ba135017de1f898ba9d40061ec25d44fb52d"} err="failed to get container status \"62db54214e8204fa57c4e62a8114ba135017de1f898ba9d40061ec25d44fb52d\": rpc error: code = NotFound desc = could not find container \"62db54214e8204fa57c4e62a8114ba135017de1f898ba9d40061ec25d44fb52d\": container with ID starting with 62db54214e8204fa57c4e62a8114ba135017de1f898ba9d40061ec25d44fb52d not found: ID does not exist"
Nov 02 09:58:42.463014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.462985    4449 scope.go:122] "RemoveContainer" containerID="f458b4a3b282515d2613beb5e7dc1d805f82b60b967d39b0ab0b937410cdeb00"
Nov 02 09:58:42.463274 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:42.463255    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f458b4a3b282515d2613beb5e7dc1d805f82b60b967d39b0ab0b937410cdeb00\": container with ID starting with f458b4a3b282515d2613beb5e7dc1d805f82b60b967d39b0ab0b937410cdeb00 not found: ID does not exist" containerID="f458b4a3b282515d2613beb5e7dc1d805f82b60b967d39b0ab0b937410cdeb00"
Nov 02 09:58:42.463344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.463278    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f458b4a3b282515d2613beb5e7dc1d805f82b60b967d39b0ab0b937410cdeb00"} err="failed to get container status \"f458b4a3b282515d2613beb5e7dc1d805f82b60b967d39b0ab0b937410cdeb00\": rpc error: code = NotFound desc = could not find container \"f458b4a3b282515d2613beb5e7dc1d805f82b60b967d39b0ab0b937410cdeb00\": container with ID starting with f458b4a3b282515d2613beb5e7dc1d805f82b60b967d39b0ab0b937410cdeb00 not found: ID does not exist"
Nov 02 09:58:42.463344 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.463294    4449 scope.go:122] "RemoveContainer" containerID="45373cd1ec8060d63bf6c6f42240f34ed8de3daab98beb28700607f02cb0ff4f"
Nov 02 09:58:42.474685 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.474669    4449 scope.go:122] "RemoveContainer" containerID="45373cd1ec8060d63bf6c6f42240f34ed8de3daab98beb28700607f02cb0ff4f"
Nov 02 09:58:42.474945 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:42.474927    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45373cd1ec8060d63bf6c6f42240f34ed8de3daab98beb28700607f02cb0ff4f\": container with ID starting with 45373cd1ec8060d63bf6c6f42240f34ed8de3daab98beb28700607f02cb0ff4f not found: ID does not exist" containerID="45373cd1ec8060d63bf6c6f42240f34ed8de3daab98beb28700607f02cb0ff4f"
Nov 02 09:58:42.474997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.474952    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45373cd1ec8060d63bf6c6f42240f34ed8de3daab98beb28700607f02cb0ff4f"} err="failed to get container status \"45373cd1ec8060d63bf6c6f42240f34ed8de3daab98beb28700607f02cb0ff4f\": rpc error: code = NotFound desc = could not find container \"45373cd1ec8060d63bf6c6f42240f34ed8de3daab98beb28700607f02cb0ff4f\": container with ID starting with 45373cd1ec8060d63bf6c6f42240f34ed8de3daab98beb28700607f02cb0ff4f not found: ID does not exist"
Nov 02 09:58:42.475979 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.475958    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"]
Nov 02 09:58:42.480135 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.480115    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-wk5lx"]
Nov 02 09:58:42.483190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.483172    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bccd9c-ca7e-4f26-81da-3d5d77be08df-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.489997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.489964    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"]
Nov 02 09:58:42.494257 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.494190    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-zx7kj"]
Nov 02 09:58:42.531364 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.531292    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"]
Nov 02 09:58:42.549037 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:42.549003    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"]
Nov 02 09:58:43.209263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:43.209233    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" path="/var/lib/kubelet/pods/2ab2b6c8-fd73-43f8-811c-d31f347557bf/volumes"
Nov 02 09:58:43.209808 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:43.209786    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" path="/var/lib/kubelet/pods/a0a81758-9db8-4221-9ab7-7a20d2400a2d/volumes"
Nov 02 09:58:43.210574 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:43.210554    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" path="/var/lib/kubelet/pods/a1bccd9c-ca7e-4f26-81da-3d5d77be08df/volumes"
Nov 02 09:58:43.211388 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:43.211349    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdbc7509-08cd-45ab-ad9d-32527a7c48af" path="/var/lib/kubelet/pods/bdbc7509-08cd-45ab-ad9d-32527a7c48af/volumes"
Nov 02 09:58:44.803417 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.803379    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"]
Nov 02 09:58:44.804024 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804010    4449 cpu_manager.go:414] "containerMap: removing container" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerName="istio-proxy"
Nov 02 09:58:44.804077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804025    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerName="istio-proxy"
Nov 02 09:58:44.804077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804047    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerName="istio-proxy"
Nov 02 09:58:44.804077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804052    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerName="istio-proxy"
Nov 02 09:58:44.804077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804061    4449 cpu_manager.go:414] "containerMap: removing container" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerName="istio-proxy"
Nov 02 09:58:44.804077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804067    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerName="istio-proxy"
Nov 02 09:58:44.804077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804072    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerName="istio-proxy"
Nov 02 09:58:44.804077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804077    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerName="istio-proxy"
Nov 02 09:58:44.804391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804088    4449 cpu_manager.go:414] "containerMap: removing container" podUID="776a2c05-d364-450b-87ed-fbcf7554a5c2" containerName="revoke-keys"
Nov 02 09:58:44.804391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804092    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776a2c05-d364-450b-87ed-fbcf7554a5c2" containerName="revoke-keys"
Nov 02 09:58:44.804391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804104    4449 cpu_manager.go:414] "containerMap: removing container" podUID="bdbc7509-08cd-45ab-ad9d-32527a7c48af" containerName="istio-proxy"
Nov 02 09:58:44.804391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804108    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdbc7509-08cd-45ab-ad9d-32527a7c48af" containerName="istio-proxy"
Nov 02 09:58:44.804391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804201    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdbc7509-08cd-45ab-ad9d-32527a7c48af" containerName="istio-proxy"
Nov 02 09:58:44.804391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804231    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerName="istio-proxy"
Nov 02 09:58:44.804391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804239    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bccd9c-ca7e-4f26-81da-3d5d77be08df" containerName="istio-proxy"
Nov 02 09:58:44.804391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804247    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a81758-9db8-4221-9ab7-7a20d2400a2d" containerName="istio-proxy"
Nov 02 09:58:44.804391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.804251    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="776a2c05-d364-450b-87ed-fbcf7554a5c2" containerName="revoke-keys"
Nov 02 09:58:44.809912 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.809888    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"
Nov 02 09:58:44.820848 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.819019    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:58:44.820848 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.819656    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:58:44.820848 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.820270    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:58:44.821391 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.821370    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:58:44.823426 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.823137    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"]
Nov 02 09:58:44.919963 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.919930    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19a2d3cc-5dh7h\" (UID: \"d8f73a74-81b9-4231-938d-404a42335a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"
Nov 02 09:58:44.920124 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.920062    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19a2d3cc-5dh7h\" (UID: \"d8f73a74-81b9-4231-938d-404a42335a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"
Nov 02 09:58:44.920261 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:44.920244    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-988w9\" (UniqueName: \"kubernetes.io/projected/d8f73a74-81b9-4231-938d-404a42335a88-kube-api-access-988w9\") pod \"maas-api-revoke-sub-19a2d3cc-5dh7h\" (UID: \"d8f73a74-81b9-4231-938d-404a42335a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"
Nov 02 09:58:45.021082 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:45.021042    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19a2d3cc-5dh7h\" (UID: \"d8f73a74-81b9-4231-938d-404a42335a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"
Nov 02 09:58:45.021308 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:45.021135    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19a2d3cc-5dh7h\" (UID: \"d8f73a74-81b9-4231-938d-404a42335a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"
Nov 02 09:58:45.021414 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:45.021312    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-988w9\" (UniqueName: \"kubernetes.io/projected/d8f73a74-81b9-4231-938d-404a42335a88-kube-api-access-988w9\") pod \"maas-api-revoke-sub-19a2d3cc-5dh7h\" (UID: \"d8f73a74-81b9-4231-938d-404a42335a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"
Nov 02 09:58:45.021788 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:45.021765    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19a2d3cc-5dh7h\" (UID: \"d8f73a74-81b9-4231-938d-404a42335a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"
Nov 02 09:58:45.023719 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:45.023693    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19a2d3cc-5dh7h\" (UID: \"d8f73a74-81b9-4231-938d-404a42335a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"
Nov 02 09:58:45.034270 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:45.034209    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-988w9\" (UniqueName: \"kubernetes.io/projected/d8f73a74-81b9-4231-938d-404a42335a88-kube-api-access-988w9\") pod \"maas-api-revoke-sub-19a2d3cc-5dh7h\" (UID: \"d8f73a74-81b9-4231-938d-404a42335a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"
Nov 02 09:58:45.153630 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:45.153542    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"
Nov 02 09:58:45.447860 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:58:45.447689    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f73a74_81b9_4231_938d_404a42335a88.slice/crio-5864b487f475412901add6b9885182be5da93e429dbdd5d9bea1ae0e3c1dfbf3 WatchSource:0}: Error finding container 5864b487f475412901add6b9885182be5da93e429dbdd5d9bea1ae0e3c1dfbf3: Status 404 returned error can't find the container with id 5864b487f475412901add6b9885182be5da93e429dbdd5d9bea1ae0e3c1dfbf3
Nov 02 09:58:45.449371 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:45.449346    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"]
Nov 02 09:58:46.468241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:46.467616    4449 generic.go:356] "Generic (PLEG): container finished" podID="d8f73a74-81b9-4231-938d-404a42335a88" containerID="e1c8add17351a72c6f5d1e22e8c1d8d3778cf28b8597834a626df76424a727ab" exitCode=0
Nov 02 09:58:46.468241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:46.467943    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h" event={"ID":"d8f73a74-81b9-4231-938d-404a42335a88","Type":"ContainerDied","Data":"e1c8add17351a72c6f5d1e22e8c1d8d3778cf28b8597834a626df76424a727ab"}
Nov 02 09:58:46.468241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:46.467995    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h" event={"ID":"d8f73a74-81b9-4231-938d-404a42335a88","Type":"ContainerStarted","Data":"5864b487f475412901add6b9885182be5da93e429dbdd5d9bea1ae0e3c1dfbf3"}
Nov 02 09:58:47.491501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.491414    4449 generic.go:356] "Generic (PLEG): container finished" podID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerID="eee7d9cba4cfe5574e4893f6da68f0097b6872d95745e4271778dc19318463cc" exitCode=0
Nov 02 09:58:47.492186 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.492025    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" event={"ID":"09f9a311-4b08-4384-8fbe-2b641f15880b","Type":"ContainerDied","Data":"eee7d9cba4cfe5574e4893f6da68f0097b6872d95745e4271778dc19318463cc"}
Nov 02 09:58:47.495974 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.495797    4449 generic.go:356] "Generic (PLEG): container finished" podID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerID="edb6b950f6cf882054c331f8062ef962daffd73a3554403e7b35df3addf5ff75" exitCode=0
Nov 02 09:58:47.495974 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.495882    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" event={"ID":"bcfda7a4-ba6a-4941-9b13-a79d554fa88a","Type":"ContainerDied","Data":"edb6b950f6cf882054c331f8062ef962daffd73a3554403e7b35df3addf5ff75"}
Nov 02 09:58:47.495974 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.495930    4449 scope.go:122] "RemoveContainer" containerID="eceb51ab9f2e960822f1c4ac856d6ee28714fa0e15601f108be40d068a7ba3cb"
Nov 02 09:58:47.502790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.502731    4449 generic.go:356] "Generic (PLEG): container finished" podID="20132bdb-f0fa-4408-9a26-5b34c1510038" containerID="e6f02ebcdcac78aceffa25695b576b3034ffcb0aa94746384ed7bd0bcd6a3ac5" exitCode=0
Nov 02 09:58:47.503107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.503053    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" event={"ID":"20132bdb-f0fa-4408-9a26-5b34c1510038","Type":"ContainerDied","Data":"e6f02ebcdcac78aceffa25695b576b3034ffcb0aa94746384ed7bd0bcd6a3ac5"}
Nov 02 09:58:47.532205 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.532162    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds" podUID="b57143d3-3a7a-4aa0-84e5-09e0b155a963" containerName="payload-processing" containerID="cri-o://41b7b48494329bf7d1e9b3a666a856a1077cf0876c94bb6064d071d0a77781a1" gracePeriod=25
Nov 02 09:58:47.533146 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.533122    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:58:47.552415 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.552370    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp" podUID="c419bca8-6a23-4b7d-bfa1-c980a92c2e08" containerName="payload-pre-processing" containerID="cri-o://8f459152acbd27aea887ab3a5637230d25782dd6051a6c5ca6b80ba7d8f09a0f" gracePeriod=25
Nov 02 09:58:47.556390 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.556362    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:58:47.615284 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.615249    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:58:47.659982 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.659949    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-kube-api-access-z4w6w\" (UniqueName: \"kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-kube-api-access-z4w6w\") pod \"09f9a311-4b08-4384-8fbe-2b641f15880b\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") "
Nov 02 09:58:47.660160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660008    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-kube-api-access-cwgzk\" (UniqueName: \"kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-kube-api-access-cwgzk\") pod \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") "
Nov 02 09:58:47.660160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660049    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-certs\") pod \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") "
Nov 02 09:58:47.660160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660080    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-data\") pod \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") "
Nov 02 09:58:47.660160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660111    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-certs\") pod \"09f9a311-4b08-4384-8fbe-2b641f15880b\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") "
Nov 02 09:58:47.660422 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660181    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-data\") pod \"09f9a311-4b08-4384-8fbe-2b641f15880b\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") "
Nov 02 09:58:47.660422 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660239    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-socket\") pod \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") "
Nov 02 09:58:47.660422 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660273    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-envoy\") pod \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") "
Nov 02 09:58:47.660422 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660309    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-credential-socket\") pod \"09f9a311-4b08-4384-8fbe-2b641f15880b\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") "
Nov 02 09:58:47.660422 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660351    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-podinfo\") pod \"09f9a311-4b08-4384-8fbe-2b641f15880b\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") "
Nov 02 09:58:47.660422 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660379    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-envoy\") pod \"09f9a311-4b08-4384-8fbe-2b641f15880b\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") "
Nov 02 09:58:47.660422 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660420    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istiod-ca-cert\") pod \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") "
Nov 02 09:58:47.660810 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660488    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-token\" (UniqueName: \"kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-token\") pod \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") "
Nov 02 09:58:47.660810 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660526    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f9a311-4b08-4384-8fbe-2b641f15880b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f9a311-4b08-4384-8fbe-2b641f15880b-istiod-ca-cert\") pod \"09f9a311-4b08-4384-8fbe-2b641f15880b\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") "
Nov 02 09:58:47.660810 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660589    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-socket\") pod \"09f9a311-4b08-4384-8fbe-2b641f15880b\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") "
Nov 02 09:58:47.660810 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660754    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-podinfo\") pod \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") "
Nov 02 09:58:47.660810 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660791    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-token\" (UniqueName: \"kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-token\") pod \"09f9a311-4b08-4384-8fbe-2b641f15880b\" (UID: \"09f9a311-4b08-4384-8fbe-2b641f15880b\") "
Nov 02 09:58:47.661077 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.660837    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-credential-socket\") pod \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\" (UID: \"bcfda7a4-ba6a-4941-9b13-a79d554fa88a\") "
Nov 02 09:58:47.661847 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.661635    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-credential-socket" pod "09f9a311-4b08-4384-8fbe-2b641f15880b" (UID: "09f9a311-4b08-4384-8fbe-2b641f15880b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.662001 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.661891    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-credential-socket" pod "bcfda7a4-ba6a-4941-9b13-a79d554fa88a" (UID: "bcfda7a4-ba6a-4941-9b13-a79d554fa88a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.662241 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.662085    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-socket" pod "09f9a311-4b08-4384-8fbe-2b641f15880b" (UID: "09f9a311-4b08-4384-8fbe-2b641f15880b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.662858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.662391    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f9a311-4b08-4384-8fbe-2b641f15880b-istiod-ca-cert" pod "09f9a311-4b08-4384-8fbe-2b641f15880b" (UID: "09f9a311-4b08-4384-8fbe-2b641f15880b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:47.662858 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.662695    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-certs" pod "09f9a311-4b08-4384-8fbe-2b641f15880b" (UID: "09f9a311-4b08-4384-8fbe-2b641f15880b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.663860 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.663163    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istiod-ca-cert" pod "bcfda7a4-ba6a-4941-9b13-a79d554fa88a" (UID: "bcfda7a4-ba6a-4941-9b13-a79d554fa88a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:47.664285 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.663517    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-certs" pod "bcfda7a4-ba6a-4941-9b13-a79d554fa88a" (UID: "bcfda7a4-ba6a-4941-9b13-a79d554fa88a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.665772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.664431    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-data" pod "bcfda7a4-ba6a-4941-9b13-a79d554fa88a" (UID: "bcfda7a4-ba6a-4941-9b13-a79d554fa88a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.665772 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.664796    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-socket" pod "bcfda7a4-ba6a-4941-9b13-a79d554fa88a" (UID: "bcfda7a4-ba6a-4941-9b13-a79d554fa88a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.666023 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.665992    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-data" pod "09f9a311-4b08-4384-8fbe-2b641f15880b" (UID: "09f9a311-4b08-4384-8fbe-2b641f15880b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.666501 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.666476    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-podinfo" pod "bcfda7a4-ba6a-4941-9b13-a79d554fa88a" (UID: "bcfda7a4-ba6a-4941-9b13-a79d554fa88a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:47.666820 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.666704    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-envoy" pod "09f9a311-4b08-4384-8fbe-2b641f15880b" (UID: "09f9a311-4b08-4384-8fbe-2b641f15880b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.666820 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.666771    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-podinfo" pod "09f9a311-4b08-4384-8fbe-2b641f15880b" (UID: "09f9a311-4b08-4384-8fbe-2b641f15880b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:47.666972 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.666820    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-kube-api-access-z4w6w" pod "09f9a311-4b08-4384-8fbe-2b641f15880b" (UID: "09f9a311-4b08-4384-8fbe-2b641f15880b"). InnerVolumeSpecName "kube-api-access-z4w6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:47.667185 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.667144    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-token" pod "09f9a311-4b08-4384-8fbe-2b641f15880b" (UID: "09f9a311-4b08-4384-8fbe-2b641f15880b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:47.667544 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.667525    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-kube-api-access-cwgzk" pod "bcfda7a4-ba6a-4941-9b13-a79d554fa88a" (UID: "bcfda7a4-ba6a-4941-9b13-a79d554fa88a"). InnerVolumeSpecName "kube-api-access-cwgzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:47.668013 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.667988    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-token" pod "bcfda7a4-ba6a-4941-9b13-a79d554fa88a" (UID: "bcfda7a4-ba6a-4941-9b13-a79d554fa88a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:47.668149 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.668130    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-envoy" pod "bcfda7a4-ba6a-4941-9b13-a79d554fa88a" (UID: "bcfda7a4-ba6a-4941-9b13-a79d554fa88a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.719665 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.719636    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"
Nov 02 09:58:47.762782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.762689    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-credential-socket\") pod \"20132bdb-f0fa-4408-9a26-5b34c1510038\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") "
Nov 02 09:58:47.762966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.762881    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-data\") pod \"20132bdb-f0fa-4408-9a26-5b34c1510038\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") "
Nov 02 09:58:47.762966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.762933    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20132bdb-f0fa-4408-9a26-5b34c1510038-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20132bdb-f0fa-4408-9a26-5b34c1510038-istiod-ca-cert\") pod \"20132bdb-f0fa-4408-9a26-5b34c1510038\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") "
Nov 02 09:58:47.762966 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.762961    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-envoy\") pod \"20132bdb-f0fa-4408-9a26-5b34c1510038\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") "
Nov 02 09:58:47.763160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.763000    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-credential-socket" pod "20132bdb-f0fa-4408-9a26-5b34c1510038" (UID: "20132bdb-f0fa-4408-9a26-5b34c1510038"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.763160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.763036    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-kube-api-access-tzqdw\" (UniqueName: \"kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-kube-api-access-tzqdw\") pod \"20132bdb-f0fa-4408-9a26-5b34c1510038\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") "
Nov 02 09:58:47.763339 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.763188    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-socket\") pod \"20132bdb-f0fa-4408-9a26-5b34c1510038\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") "
Nov 02 09:58:47.763339 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.763252    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-certs\") pod \"20132bdb-f0fa-4408-9a26-5b34c1510038\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") "
Nov 02 09:58:47.763339 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.763286    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-token\" (UniqueName: \"kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-token\") pod \"20132bdb-f0fa-4408-9a26-5b34c1510038\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") "
Nov 02 09:58:47.763339 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.763320    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-podinfo\") pod \"20132bdb-f0fa-4408-9a26-5b34c1510038\" (UID: \"20132bdb-f0fa-4408-9a26-5b34c1510038\") "
Nov 02 09:58:47.763636 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.763331    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-data" pod "20132bdb-f0fa-4408-9a26-5b34c1510038" (UID: "20132bdb-f0fa-4408-9a26-5b34c1510038"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.763636 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.763428    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20132bdb-f0fa-4408-9a26-5b34c1510038-istiod-ca-cert" pod "20132bdb-f0fa-4408-9a26-5b34c1510038" (UID: "20132bdb-f0fa-4408-9a26-5b34c1510038"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:47.763818 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.763791    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-socket" pod "20132bdb-f0fa-4408-9a26-5b34c1510038" (UID: "20132bdb-f0fa-4408-9a26-5b34c1510038"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.763933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.763908    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-certs" pod "20132bdb-f0fa-4408-9a26-5b34c1510038" (UID: "20132bdb-f0fa-4408-9a26-5b34c1510038"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.764008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.763958    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.763980    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.763995    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764008    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764022    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20132bdb-f0fa-4408-9a26-5b34c1510038-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764036    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4w6w\" (UniqueName: \"kubernetes.io/projected/09f9a311-4b08-4384-8fbe-2b641f15880b-kube-api-access-z4w6w\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764053    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwgzk\" (UniqueName: \"kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-kube-api-access-cwgzk\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764066    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764079    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764091    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764106    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764123    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764137    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764152    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764173 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764168    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764181    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764195    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764209    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764242    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcfda7a4-ba6a-4941-9b13-a79d554fa88a-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764256    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f9a311-4b08-4384-8fbe-2b641f15880b-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764270    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.764933 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.764283    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f9a311-4b08-4384-8fbe-2b641f15880b-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.766878 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.766843    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-kube-api-access-tzqdw" pod "20132bdb-f0fa-4408-9a26-5b34c1510038" (UID: "20132bdb-f0fa-4408-9a26-5b34c1510038"). InnerVolumeSpecName "kube-api-access-tzqdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:47.767001 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.766938    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-podinfo" pod "20132bdb-f0fa-4408-9a26-5b34c1510038" (UID: "20132bdb-f0fa-4408-9a26-5b34c1510038"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:47.767001 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.766983    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-token" pod "20132bdb-f0fa-4408-9a26-5b34c1510038" (UID: "20132bdb-f0fa-4408-9a26-5b34c1510038"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:47.768495 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.768438    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-envoy" pod "20132bdb-f0fa-4408-9a26-5b34c1510038" (UID: "20132bdb-f0fa-4408-9a26-5b34c1510038"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.865395 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.865358    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-openshift-ca\") pod \"d8f73a74-81b9-4231-938d-404a42335a88\" (UID: \"d8f73a74-81b9-4231-938d-404a42335a88\") "
Nov 02 09:58:47.865554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.865477    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f73a74-81b9-4231-938d-404a42335a88-kube-api-access-988w9\" (UniqueName: \"kubernetes.io/projected/d8f73a74-81b9-4231-938d-404a42335a88-kube-api-access-988w9\") pod \"d8f73a74-81b9-4231-938d-404a42335a88\" (UID: \"d8f73a74-81b9-4231-938d-404a42335a88\") "
Nov 02 09:58:47.865554 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.865506    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-xks-ca\") pod \"d8f73a74-81b9-4231-938d-404a42335a88\" (UID: \"d8f73a74-81b9-4231-938d-404a42335a88\") "
Nov 02 09:58:47.865737 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.865705    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-openshift-ca" pod "d8f73a74-81b9-4231-938d-404a42335a88" (UID: "d8f73a74-81b9-4231-938d-404a42335a88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:47.865958 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.865935    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.866033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.865960    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.866033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.865971    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzqdw\" (UniqueName: \"kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-kube-api-access-tzqdw\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.866033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.865980    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20132bdb-f0fa-4408-9a26-5b34c1510038-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.866033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.865988    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.866033 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.865996    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20132bdb-f0fa-4408-9a26-5b34c1510038-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.867810 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.867783    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-xks-ca" pod "d8f73a74-81b9-4231-938d-404a42335a88" (UID: "d8f73a74-81b9-4231-938d-404a42335a88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:58:47.867928 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.867900    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f73a74-81b9-4231-938d-404a42335a88-kube-api-access-988w9" pod "d8f73a74-81b9-4231-938d-404a42335a88" (UID: "d8f73a74-81b9-4231-938d-404a42335a88"). InnerVolumeSpecName "kube-api-access-988w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:47.967444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.967406    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-988w9\" (UniqueName: \"kubernetes.io/projected/d8f73a74-81b9-4231-938d-404a42335a88-kube-api-access-988w9\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.967444 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:47.967439    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f73a74-81b9-4231-938d-404a42335a88-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:48.094560 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.094515    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp" podUID="c419bca8-6a23-4b7d-bfa1-c980a92c2e08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:58:48.514263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.514161    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"
Nov 02 09:58:48.514263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.514196    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h" event={"ID":"d8f73a74-81b9-4231-938d-404a42335a88","Type":"ContainerDied","Data":"5864b487f475412901add6b9885182be5da93e429dbdd5d9bea1ae0e3c1dfbf3"}
Nov 02 09:58:48.514263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.514259    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5864b487f475412901add6b9885182be5da93e429dbdd5d9bea1ae0e3c1dfbf3"
Nov 02 09:58:48.515927 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.515903    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"
Nov 02 09:58:48.516112 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.515902    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq" event={"ID":"09f9a311-4b08-4384-8fbe-2b641f15880b","Type":"ContainerDied","Data":"bf5e0e1017b81b15ac126d5c46754c36abf3491d2817ac2d9eaf5139aff42bf9"}
Nov 02 09:58:48.516112 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.516037    4449 scope.go:122] "RemoveContainer" containerID="eee7d9cba4cfe5574e4893f6da68f0097b6872d95745e4271778dc19318463cc"
Nov 02 09:58:48.527875 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.527637    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq" event={"ID":"bcfda7a4-ba6a-4941-9b13-a79d554fa88a","Type":"ContainerDied","Data":"13ce58fbeb4ce2d6c1ef45a3579f0f41c2fa15531f34b13440a52e903330b18c"}
Nov 02 09:58:48.527875 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.527669    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"
Nov 02 09:58:48.529782 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.529742    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6" event={"ID":"20132bdb-f0fa-4408-9a26-5b34c1510038","Type":"ContainerDied","Data":"08e948fd33380a7b5b3cd5b3d37ddd1a39e73894a72710441a9c2c7ce648155c"}
Nov 02 09:58:48.529967 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.529948    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"
Nov 02 09:58:48.541606 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.541582    4449 scope.go:122] "RemoveContainer" containerID="edb6b950f6cf882054c331f8062ef962daffd73a3554403e7b35df3addf5ff75"
Nov 02 09:58:48.552995 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.552971    4449 scope.go:122] "RemoveContainer" containerID="e6f02ebcdcac78aceffa25695b576b3034ffcb0aa94746384ed7bd0bcd6a3ac5"
Nov 02 09:58:48.569800 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.569774    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"]
Nov 02 09:58:48.575833 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.575809    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-6krb6"]
Nov 02 09:58:48.589702 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.589673    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"]
Nov 02 09:58:48.596516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.596486    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-gkljq"]
Nov 02 09:58:48.625619 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.625589    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"]
Nov 02 09:58:48.629724 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.629696    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-sk5vq"]
Nov 02 09:58:48.820856 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:48.820817    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk" podUID="35d1f11e-bd29-4c15-af90-027c65c690fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:58:49.112790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:49.112706    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds" podUID="b57143d3-3a7a-4aa0-84e5-09e0b155a963" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:58:49.206849 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:49.206810    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" path="/var/lib/kubelet/pods/09f9a311-4b08-4384-8fbe-2b641f15880b/volumes"
Nov 02 09:58:49.207429 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:49.207406    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20132bdb-f0fa-4408-9a26-5b34c1510038" path="/var/lib/kubelet/pods/20132bdb-f0fa-4408-9a26-5b34c1510038/volumes"
Nov 02 09:58:49.207935 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:49.207915    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" path="/var/lib/kubelet/pods/bcfda7a4-ba6a-4941-9b13-a79d554fa88a/volumes"
Nov 02 09:58:49.826167 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:49.826131    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t" podUID="7fcc4135-f9c4-4c04-96da-26ad2abd502e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:58:52.646663 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:52.643673    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"]
Nov 02 09:58:52.646663 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:52.643957    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm" podUID="f5071cf6-026d-407b-84b3-f682c15a8d22" containerName="istio-proxy" containerID="cri-o://19c2cba41a689ece661d978de644dc68f5e8763df72fb8f2e0381df12911d14d" gracePeriod=30
Nov 02 09:58:52.653582 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:52.652481    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"]
Nov 02 09:58:52.653582 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:52.652727    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy" containerID="cri-o://2a00e9bcc65ad9494ec7b859ae35cd64b37b4985c196c0e9f6857d02d9fd5e86" gracePeriod=30
Nov 02 09:58:52.664401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:52.658232    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"]
Nov 02 09:58:52.664401 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:52.658489    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f" podUID="220a9b39-e990-44ed-bdf7-4c039fa4a9ec" containerName="istio-proxy" containerID="cri-o://dc48825f6451ccdc587ee88c6d5c207a70a0b4deefadf8dd3566e45e84f40832" gracePeriod=30
Nov 02 09:58:52.667197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:52.665169    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"]
Nov 02 09:58:52.667197 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:52.665388    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" podUID="11c2db3f-6464-47e8-af64-2c432474e832" containerName="istio-proxy" containerID="cri-o://20bc5977a89804e49285c5d35b26f3dd0ff7a62db9534c2336e1c4914add7854" gracePeriod=30
Nov 02 09:58:53.898202 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:53.898173    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"
Nov 02 09:58:53.945502 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:53.945470    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"
Nov 02 09:58:54.024736 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.024698    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fcc4135-f9c4-4c04-96da-26ad2abd502e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcc4135-f9c4-4c04-96da-26ad2abd502e-plugins-config-volume\") pod \"7fcc4135-f9c4-4c04-96da-26ad2abd502e\" (UID: \"7fcc4135-f9c4-4c04-96da-26ad2abd502e\") "
Nov 02 09:58:54.024917 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.024753    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fcc4135-f9c4-4c04-96da-26ad2abd502e-kube-api-access-q7lr4\" (UniqueName: \"kubernetes.io/projected/7fcc4135-f9c4-4c04-96da-26ad2abd502e-kube-api-access-q7lr4\") pod \"7fcc4135-f9c4-4c04-96da-26ad2abd502e\" (UID: \"7fcc4135-f9c4-4c04-96da-26ad2abd502e\") "
Nov 02 09:58:54.024917 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.024773    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35d1f11e-bd29-4c15-af90-027c65c690fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d1f11e-bd29-4c15-af90-027c65c690fa-plugins-config-volume\") pod \"35d1f11e-bd29-4c15-af90-027c65c690fa\" (UID: \"35d1f11e-bd29-4c15-af90-027c65c690fa\") "
Nov 02 09:58:54.024917 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.024818    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35d1f11e-bd29-4c15-af90-027c65c690fa-kube-api-access-kvntd\" (UniqueName: \"kubernetes.io/projected/35d1f11e-bd29-4c15-af90-027c65c690fa-kube-api-access-kvntd\") pod \"35d1f11e-bd29-4c15-af90-027c65c690fa\" (UID: \"35d1f11e-bd29-4c15-af90-027c65c690fa\") "
Nov 02 09:58:54.025163 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.025141    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d1f11e-bd29-4c15-af90-027c65c690fa-plugins-config-volume" pod "35d1f11e-bd29-4c15-af90-027c65c690fa" (UID: "35d1f11e-bd29-4c15-af90-027c65c690fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:54.025205 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.025161    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fcc4135-f9c4-4c04-96da-26ad2abd502e-plugins-config-volume" pod "7fcc4135-f9c4-4c04-96da-26ad2abd502e" (UID: "7fcc4135-f9c4-4c04-96da-26ad2abd502e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:54.025354 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.025335    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcc4135-f9c4-4c04-96da-26ad2abd502e-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:54.025410 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.025357    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35d1f11e-bd29-4c15-af90-027c65c690fa-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:54.027156 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.027134    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fcc4135-f9c4-4c04-96da-26ad2abd502e-kube-api-access-q7lr4" pod "7fcc4135-f9c4-4c04-96da-26ad2abd502e" (UID: "7fcc4135-f9c4-4c04-96da-26ad2abd502e"). InnerVolumeSpecName "kube-api-access-q7lr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:54.027156 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.027147    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d1f11e-bd29-4c15-af90-027c65c690fa-kube-api-access-kvntd" pod "35d1f11e-bd29-4c15-af90-027c65c690fa" (UID: "35d1f11e-bd29-4c15-af90-027c65c690fa"). InnerVolumeSpecName "kube-api-access-kvntd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:54.127226 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.127119    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7lr4\" (UniqueName: \"kubernetes.io/projected/7fcc4135-f9c4-4c04-96da-26ad2abd502e-kube-api-access-q7lr4\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:54.127226 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.127151    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvntd\" (UniqueName: \"kubernetes.io/projected/35d1f11e-bd29-4c15-af90-027c65c690fa-kube-api-access-kvntd\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:54.583156 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.583126    4449 generic.go:356] "Generic (PLEG): container finished" podID="7fcc4135-f9c4-4c04-96da-26ad2abd502e" containerID="3156aa548a2904e0ade7d04c4d901bc6de93f99bc34a66c365da3a0b07de8673" exitCode=0
Nov 02 09:58:54.583385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.583169    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"
Nov 02 09:58:54.583385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.583212    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t" event={"ID":"7fcc4135-f9c4-4c04-96da-26ad2abd502e","Type":"ContainerDied","Data":"3156aa548a2904e0ade7d04c4d901bc6de93f99bc34a66c365da3a0b07de8673"}
Nov 02 09:58:54.583385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.583278    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t" event={"ID":"7fcc4135-f9c4-4c04-96da-26ad2abd502e","Type":"ContainerDied","Data":"a7418bbe5552f0c37a05887a2ff121ca017de86a707340c9808d8256b81499f9"}
Nov 02 09:58:54.583385 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.583294    4449 scope.go:122] "RemoveContainer" containerID="3156aa548a2904e0ade7d04c4d901bc6de93f99bc34a66c365da3a0b07de8673"
Nov 02 09:58:54.584911 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.584888    4449 generic.go:356] "Generic (PLEG): container finished" podID="35d1f11e-bd29-4c15-af90-027c65c690fa" containerID="e3dfe6a5133d381ba3b385b7243890b474b52df8d1898892013a1ea77d4d146d" exitCode=0
Nov 02 09:58:54.585095 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.585065    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk" event={"ID":"35d1f11e-bd29-4c15-af90-027c65c690fa","Type":"ContainerDied","Data":"e3dfe6a5133d381ba3b385b7243890b474b52df8d1898892013a1ea77d4d146d"}
Nov 02 09:58:54.585199 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.585097    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"
Nov 02 09:58:54.585199 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.585115    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk" event={"ID":"35d1f11e-bd29-4c15-af90-027c65c690fa","Type":"ContainerDied","Data":"7ed5adf4d564e91038f9678ba1eb5cfaa4b04e491f0ab936b79ab57f92cecc99"}
Nov 02 09:58:54.600305 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.600282    4449 scope.go:122] "RemoveContainer" containerID="3156aa548a2904e0ade7d04c4d901bc6de93f99bc34a66c365da3a0b07de8673"
Nov 02 09:58:54.603563 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:54.603536    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3156aa548a2904e0ade7d04c4d901bc6de93f99bc34a66c365da3a0b07de8673\": container with ID starting with 3156aa548a2904e0ade7d04c4d901bc6de93f99bc34a66c365da3a0b07de8673 not found: ID does not exist" containerID="3156aa548a2904e0ade7d04c4d901bc6de93f99bc34a66c365da3a0b07de8673"
Nov 02 09:58:54.603685 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.603567    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3156aa548a2904e0ade7d04c4d901bc6de93f99bc34a66c365da3a0b07de8673"} err="failed to get container status \"3156aa548a2904e0ade7d04c4d901bc6de93f99bc34a66c365da3a0b07de8673\": rpc error: code = NotFound desc = could not find container \"3156aa548a2904e0ade7d04c4d901bc6de93f99bc34a66c365da3a0b07de8673\": container with ID starting with 3156aa548a2904e0ade7d04c4d901bc6de93f99bc34a66c365da3a0b07de8673 not found: ID does not exist"
Nov 02 09:58:54.603685 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.603586    4449 scope.go:122] "RemoveContainer" containerID="e3dfe6a5133d381ba3b385b7243890b474b52df8d1898892013a1ea77d4d146d"
Nov 02 09:58:54.618852 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.618822    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"]
Nov 02 09:58:54.624700 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.624675    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-bde944-99767bc7f-kbcnk"]
Nov 02 09:58:54.631440 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.631423    4449 scope.go:122] "RemoveContainer" containerID="e3dfe6a5133d381ba3b385b7243890b474b52df8d1898892013a1ea77d4d146d"
Nov 02 09:58:54.631729 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:54.631705    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3dfe6a5133d381ba3b385b7243890b474b52df8d1898892013a1ea77d4d146d\": container with ID starting with e3dfe6a5133d381ba3b385b7243890b474b52df8d1898892013a1ea77d4d146d not found: ID does not exist" containerID="e3dfe6a5133d381ba3b385b7243890b474b52df8d1898892013a1ea77d4d146d"
Nov 02 09:58:54.631828 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.631740    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3dfe6a5133d381ba3b385b7243890b474b52df8d1898892013a1ea77d4d146d"} err="failed to get container status \"e3dfe6a5133d381ba3b385b7243890b474b52df8d1898892013a1ea77d4d146d\": rpc error: code = NotFound desc = could not find container \"e3dfe6a5133d381ba3b385b7243890b474b52df8d1898892013a1ea77d4d146d\": container with ID starting with e3dfe6a5133d381ba3b385b7243890b474b52df8d1898892013a1ea77d4d146d not found: ID does not exist"
Nov 02 09:58:54.635196 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.635174    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"]
Nov 02 09:58:54.639208 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:54.639188    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-bde944-c67f95877-bfj8t"]
Nov 02 09:58:55.025628 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:55.025542    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"]
Nov 02 09:58:55.029576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:55.029546    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30f837a-cx7pl"]
Nov 02 09:58:55.207935 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:55.207897    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336e15ff-516c-4baa-bd54-82ddfd4c25cd" path="/var/lib/kubelet/pods/336e15ff-516c-4baa-bd54-82ddfd4c25cd/volumes"
Nov 02 09:58:55.208372 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:55.208358    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35d1f11e-bd29-4c15-af90-027c65c690fa" path="/var/lib/kubelet/pods/35d1f11e-bd29-4c15-af90-027c65c690fa/volumes"
Nov 02 09:58:55.208758 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:55.208744    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fcc4135-f9c4-4c04-96da-26ad2abd502e" path="/var/lib/kubelet/pods/7fcc4135-f9c4-4c04-96da-26ad2abd502e/volumes"
Nov 02 09:58:57.355655 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.355616    4449 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 09:58:57.356144 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.355681    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 09:58:57.634788 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.634713    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"]
Nov 02 09:58:57.635307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635293    4449 cpu_manager.go:414] "containerMap: removing container" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerName="istio-proxy"
Nov 02 09:58:57.635349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635308    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerName="istio-proxy"
Nov 02 09:58:57.635349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635320    4449 cpu_manager.go:414] "containerMap: removing container" podUID="d8f73a74-81b9-4231-938d-404a42335a88" containerName="revoke-keys"
Nov 02 09:58:57.635349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635326    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f73a74-81b9-4231-938d-404a42335a88" containerName="revoke-keys"
Nov 02 09:58:57.635349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635347    4449 cpu_manager.go:414] "containerMap: removing container" podUID="35d1f11e-bd29-4c15-af90-027c65c690fa" containerName="payload-pre-processing"
Nov 02 09:58:57.635489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635352    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35d1f11e-bd29-4c15-af90-027c65c690fa" containerName="payload-pre-processing"
Nov 02 09:58:57.635489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635366    4449 cpu_manager.go:414] "containerMap: removing container" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerName="istio-proxy"
Nov 02 09:58:57.635489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635371    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerName="istio-proxy"
Nov 02 09:58:57.635489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635386    4449 cpu_manager.go:414] "containerMap: removing container" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerName="istio-proxy"
Nov 02 09:58:57.635489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635392    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerName="istio-proxy"
Nov 02 09:58:57.635489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635403    4449 cpu_manager.go:414] "containerMap: removing container" podUID="7fcc4135-f9c4-4c04-96da-26ad2abd502e" containerName="payload-processing"
Nov 02 09:58:57.635489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635407    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fcc4135-f9c4-4c04-96da-26ad2abd502e" containerName="payload-processing"
Nov 02 09:58:57.635489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635421    4449 cpu_manager.go:414] "containerMap: removing container" podUID="20132bdb-f0fa-4408-9a26-5b34c1510038" containerName="istio-proxy"
Nov 02 09:58:57.635489 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635425    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20132bdb-f0fa-4408-9a26-5b34c1510038" containerName="istio-proxy"
Nov 02 09:58:57.635762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635498    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerName="istio-proxy"
Nov 02 09:58:57.635762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635507    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fcc4135-f9c4-4c04-96da-26ad2abd502e" containerName="payload-processing"
Nov 02 09:58:57.635762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635515    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="20132bdb-f0fa-4408-9a26-5b34c1510038" containerName="istio-proxy"
Nov 02 09:58:57.635762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635520    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcfda7a4-ba6a-4941-9b13-a79d554fa88a" containerName="istio-proxy"
Nov 02 09:58:57.635762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635526    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f73a74-81b9-4231-938d-404a42335a88" containerName="revoke-keys"
Nov 02 09:58:57.635762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635533    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="35d1f11e-bd29-4c15-af90-027c65c690fa" containerName="payload-pre-processing"
Nov 02 09:58:57.635762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635539    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab2b6c8-fd73-43f8-811c-d31f347557bf" containerName="istio-proxy"
Nov 02 09:58:57.635762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.635545    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f9a311-4b08-4384-8fbe-2b641f15880b" containerName="istio-proxy"
Nov 02 09:58:57.640629 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.640612    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"
Nov 02 09:58:57.646835 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.646805    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:58:57.647053 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.646859    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:58:57.649684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.649663    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"]
Nov 02 09:58:57.653774 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.653754    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:58:57.654080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.654043    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:58:57.778790 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.778765    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40f185ba-7zbgd\" (UID: \"7701534c-2a3d-4211-a06a-2af4a6b454f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"
Nov 02 09:58:57.778997 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.778903    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvnbz\" (UniqueName: \"kubernetes.io/projected/7701534c-2a3d-4211-a06a-2af4a6b454f7-kube-api-access-zvnbz\") pod \"maas-api-revoke-sub-40f185ba-7zbgd\" (UID: \"7701534c-2a3d-4211-a06a-2af4a6b454f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"
Nov 02 09:58:57.779066 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.779008    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40f185ba-7zbgd\" (UID: \"7701534c-2a3d-4211-a06a-2af4a6b454f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"
Nov 02 09:58:57.883206 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.883163    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40f185ba-7zbgd\" (UID: \"7701534c-2a3d-4211-a06a-2af4a6b454f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"
Nov 02 09:58:57.883383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.883293    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvnbz\" (UniqueName: \"kubernetes.io/projected/7701534c-2a3d-4211-a06a-2af4a6b454f7-kube-api-access-zvnbz\") pod \"maas-api-revoke-sub-40f185ba-7zbgd\" (UID: \"7701534c-2a3d-4211-a06a-2af4a6b454f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"
Nov 02 09:58:57.883457 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.883394    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40f185ba-7zbgd\" (UID: \"7701534c-2a3d-4211-a06a-2af4a6b454f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"
Nov 02 09:58:57.885010 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.884918    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40f185ba-7zbgd\" (UID: \"7701534c-2a3d-4211-a06a-2af4a6b454f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"
Nov 02 09:58:57.886510 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.886463    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40f185ba-7zbgd\" (UID: \"7701534c-2a3d-4211-a06a-2af4a6b454f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"
Nov 02 09:58:57.898882 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.898832    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvnbz\" (UniqueName: \"kubernetes.io/projected/7701534c-2a3d-4211-a06a-2af4a6b454f7-kube-api-access-zvnbz\") pod \"maas-api-revoke-sub-40f185ba-7zbgd\" (UID: \"7701534c-2a3d-4211-a06a-2af4a6b454f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"
Nov 02 09:58:57.956349 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:57.956311    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"
Nov 02 09:58:58.020910 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.020835    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:58:58.045888 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.045852    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:58:58.094944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.094915    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp" podUID="c419bca8-6a23-4b7d-bfa1-c980a92c2e08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:58:58.118127 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.117847    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:58:58.123055 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.122834    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:58:58.206463 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206378    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-certs\") pod \"0c312869-83d6-406b-bff4-6dc3068df5bf\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") "
Nov 02 09:58:58.206463 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206422    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-envoy\") pod \"0c312869-83d6-406b-bff4-6dc3068df5bf\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") "
Nov 02 09:58:58.206463 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206442    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-credential-socket\") pod \"0c312869-83d6-406b-bff4-6dc3068df5bf\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") "
Nov 02 09:58:58.206762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206479    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-istio-token\" (UniqueName: \"kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-istio-token\") pod \"11c2db3f-6464-47e8-af64-2c432474e832\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") "
Nov 02 09:58:58.206762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206502    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c312869-83d6-406b-bff4-6dc3068df5bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c312869-83d6-406b-bff4-6dc3068df5bf-istiod-ca-cert\") pod \"0c312869-83d6-406b-bff4-6dc3068df5bf\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") "
Nov 02 09:58:58.206762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206539    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-socket\") pod \"0c312869-83d6-406b-bff4-6dc3068df5bf\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") "
Nov 02 09:58:58.206762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206590    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-kube-api-access-h977d\" (UniqueName: \"kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-kube-api-access-h977d\") pod \"11c2db3f-6464-47e8-af64-2c432474e832\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") "
Nov 02 09:58:58.206762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206618    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-kube-api-access-vjt2p\" (UniqueName: \"kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-kube-api-access-vjt2p\") pod \"0c312869-83d6-406b-bff4-6dc3068df5bf\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") "
Nov 02 09:58:58.206762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206643    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-certs\") pod \"11c2db3f-6464-47e8-af64-2c432474e832\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") "
Nov 02 09:58:58.206762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206695    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-credential-socket\") pod \"11c2db3f-6464-47e8-af64-2c432474e832\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") "
Nov 02 09:58:58.206762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206736    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11c2db3f-6464-47e8-af64-2c432474e832-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c2db3f-6464-47e8-af64-2c432474e832-istio-podinfo\") pod \"11c2db3f-6464-47e8-af64-2c432474e832\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") "
Nov 02 09:58:58.207176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206801    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-podinfo\") pod \"0c312869-83d6-406b-bff4-6dc3068df5bf\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") "
Nov 02 09:58:58.207176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206861    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-data\") pod \"11c2db3f-6464-47e8-af64-2c432474e832\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") "
Nov 02 09:58:58.207176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206892    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-envoy\") pod \"11c2db3f-6464-47e8-af64-2c432474e832\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") "
Nov 02 09:58:58.207176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206929    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-socket\") pod \"11c2db3f-6464-47e8-af64-2c432474e832\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") "
Nov 02 09:58:58.207176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206948    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-socket" pod "0c312869-83d6-406b-bff4-6dc3068df5bf" (UID: "0c312869-83d6-406b-bff4-6dc3068df5bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.207176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.206960    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-data\") pod \"0c312869-83d6-406b-bff4-6dc3068df5bf\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") "
Nov 02 09:58:58.207176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.207039    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-token\" (UniqueName: \"kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-token\") pod \"0c312869-83d6-406b-bff4-6dc3068df5bf\" (UID: \"0c312869-83d6-406b-bff4-6dc3068df5bf\") "
Nov 02 09:58:58.207176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.207079    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c2db3f-6464-47e8-af64-2c432474e832-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c2db3f-6464-47e8-af64-2c432474e832-istiod-ca-cert\") pod \"11c2db3f-6464-47e8-af64-2c432474e832\" (UID: \"11c2db3f-6464-47e8-af64-2c432474e832\") "
Nov 02 09:58:58.207176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.207086    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-certs" pod "0c312869-83d6-406b-bff4-6dc3068df5bf" (UID: "0c312869-83d6-406b-bff4-6dc3068df5bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.207702 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.207334    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-credential-socket" pod "0c312869-83d6-406b-bff4-6dc3068df5bf" (UID: "0c312869-83d6-406b-bff4-6dc3068df5bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.207702 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.207361    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-certs" pod "11c2db3f-6464-47e8-af64-2c432474e832" (UID: "11c2db3f-6464-47e8-af64-2c432474e832"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.207702 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.207643    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c312869-83d6-406b-bff4-6dc3068df5bf-istiod-ca-cert" pod "0c312869-83d6-406b-bff4-6dc3068df5bf" (UID: "0c312869-83d6-406b-bff4-6dc3068df5bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:58.207945 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.207901    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c2db3f-6464-47e8-af64-2c432474e832-istiod-ca-cert" pod "11c2db3f-6464-47e8-af64-2c432474e832" (UID: "11c2db3f-6464-47e8-af64-2c432474e832"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:58.208081 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.208058    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c2db3f-6464-47e8-af64-2c432474e832-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.208158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.208090    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.208158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.208106    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.208158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.208120    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c312869-83d6-406b-bff4-6dc3068df5bf-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.208158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.208134    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.208158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.208150    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.208158 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.208170    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-credential-socket" pod "11c2db3f-6464-47e8-af64-2c432474e832" (UID: "11c2db3f-6464-47e8-af64-2c432474e832"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.208726 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.208401    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-socket" pod "11c2db3f-6464-47e8-af64-2c432474e832" (UID: "11c2db3f-6464-47e8-af64-2c432474e832"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.208726 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.208407    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-data" pod "0c312869-83d6-406b-bff4-6dc3068df5bf" (UID: "0c312869-83d6-406b-bff4-6dc3068df5bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.210689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.209592    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-istio-token" pod "11c2db3f-6464-47e8-af64-2c432474e832" (UID: "11c2db3f-6464-47e8-af64-2c432474e832"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.210689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.209859    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-data" pod "11c2db3f-6464-47e8-af64-2c432474e832" (UID: "11c2db3f-6464-47e8-af64-2c432474e832"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.210689 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.210115    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11c2db3f-6464-47e8-af64-2c432474e832-istio-podinfo" pod "11c2db3f-6464-47e8-af64-2c432474e832" (UID: "11c2db3f-6464-47e8-af64-2c432474e832"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:58.211035 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.211009    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-token" pod "0c312869-83d6-406b-bff4-6dc3068df5bf" (UID: "0c312869-83d6-406b-bff4-6dc3068df5bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.211364 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.211202    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-envoy" pod "11c2db3f-6464-47e8-af64-2c432474e832" (UID: "11c2db3f-6464-47e8-af64-2c432474e832"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.211364 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.211331    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-kube-api-access-vjt2p" pod "0c312869-83d6-406b-bff4-6dc3068df5bf" (UID: "0c312869-83d6-406b-bff4-6dc3068df5bf"). InnerVolumeSpecName "kube-api-access-vjt2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.211364 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.211334    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-envoy" pod "0c312869-83d6-406b-bff4-6dc3068df5bf" (UID: "0c312869-83d6-406b-bff4-6dc3068df5bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.211676 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.211467    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-kube-api-access-h977d" pod "11c2db3f-6464-47e8-af64-2c432474e832" (UID: "11c2db3f-6464-47e8-af64-2c432474e832"). InnerVolumeSpecName "kube-api-access-h977d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.212281 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.212261    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-podinfo" pod "0c312869-83d6-406b-bff4-6dc3068df5bf" (UID: "0c312869-83d6-406b-bff4-6dc3068df5bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:58.305709 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:58:58.305670    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7701534c_2a3d_4211_a06a_2af4a6b454f7.slice/crio-ee89834e732c50b4e3534882fa7fa08aff99ecd9006e2bc32cf4b7ab758f432f WatchSource:0}: Error finding container ee89834e732c50b4e3534882fa7fa08aff99ecd9006e2bc32cf4b7ab758f432f: Status 404 returned error can't find the container with id ee89834e732c50b4e3534882fa7fa08aff99ecd9006e2bc32cf4b7ab758f432f
Nov 02 09:58:58.306204 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.306132    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"]
Nov 02 09:58:58.308534 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.308506    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5071cf6-026d-407b-84b3-f682c15a8d22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5071cf6-026d-407b-84b3-f682c15a8d22-istiod-ca-cert\") pod \"f5071cf6-026d-407b-84b3-f682c15a8d22\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") "
Nov 02 09:58:58.308688 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.308552    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-envoy\") pod \"f5071cf6-026d-407b-84b3-f682c15a8d22\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") "
Nov 02 09:58:58.308688 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.308655    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-data\") pod \"f5071cf6-026d-407b-84b3-f682c15a8d22\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") "
Nov 02 09:58:58.308948 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.308897    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-data" pod "f5071cf6-026d-407b-84b3-f682c15a8d22" (UID: "f5071cf6-026d-407b-84b3-f682c15a8d22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.309061 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.308958    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-certs\") pod \"f5071cf6-026d-407b-84b3-f682c15a8d22\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") "
Nov 02 09:58:58.309061 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.308998    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-socket\") pod \"f5071cf6-026d-407b-84b3-f682c15a8d22\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") "
Nov 02 09:58:58.309061 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309021    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-envoy\") pod \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") "
Nov 02 09:58:58.309061 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309049    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5071cf6-026d-407b-84b3-f682c15a8d22-istiod-ca-cert" pod "f5071cf6-026d-407b-84b3-f682c15a8d22" (UID: "f5071cf6-026d-407b-84b3-f682c15a8d22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:58.309303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309057    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-kube-api-access-cg88d\" (UniqueName: \"kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-kube-api-access-cg88d\") pod \"f5071cf6-026d-407b-84b3-f682c15a8d22\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") "
Nov 02 09:58:58.309303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309139    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-podinfo\") pod \"f5071cf6-026d-407b-84b3-f682c15a8d22\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") "
Nov 02 09:58:58.309303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309268    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-data\") pod \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") "
Nov 02 09:58:58.309303 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309289    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-kube-api-access-prn5v\" (UniqueName: \"kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-kube-api-access-prn5v\") pod \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") "
Nov 02 09:58:58.309517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309314    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-token\" (UniqueName: \"kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-token\") pod \"f5071cf6-026d-407b-84b3-f682c15a8d22\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") "
Nov 02 09:58:58.309517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309375    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-certs\") pod \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") "
Nov 02 09:58:58.309517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309381    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-certs" pod "f5071cf6-026d-407b-84b3-f682c15a8d22" (UID: "f5071cf6-026d-407b-84b3-f682c15a8d22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.309517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309405    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istiod-ca-cert\") pod \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") "
Nov 02 09:58:58.309517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309463    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-credential-socket\") pod \"f5071cf6-026d-407b-84b3-f682c15a8d22\" (UID: \"f5071cf6-026d-407b-84b3-f682c15a8d22\") "
Nov 02 09:58:58.309517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309505    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-podinfo\") pod \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") "
Nov 02 09:58:58.309822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309544    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-socket\") pod \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") "
Nov 02 09:58:58.309822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309588    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-token\" (UniqueName: \"kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-token\") pod \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") "
Nov 02 09:58:58.309822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309686    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-socket" pod "f5071cf6-026d-407b-84b3-f682c15a8d22" (UID: "f5071cf6-026d-407b-84b3-f682c15a8d22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.309822 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.309782    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-credential-socket\") pod \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\" (UID: \"220a9b39-e990-44ed-bdf7-4c039fa4a9ec\") "
Nov 02 09:58:58.310431 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.310251    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-credential-socket" pod "220a9b39-e990-44ed-bdf7-4c039fa4a9ec" (UID: "220a9b39-e990-44ed-bdf7-4c039fa4a9ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.310529 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.310461    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-data" pod "220a9b39-e990-44ed-bdf7-4c039fa4a9ec" (UID: "220a9b39-e990-44ed-bdf7-4c039fa4a9ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.310769 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.310643    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-credential-socket" pod "f5071cf6-026d-407b-84b3-f682c15a8d22" (UID: "f5071cf6-026d-407b-84b3-f682c15a8d22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.311535 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.311508    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-certs" pod "220a9b39-e990-44ed-bdf7-4c039fa4a9ec" (UID: "220a9b39-e990-44ed-bdf7-4c039fa4a9ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.311650 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.311554    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istiod-ca-cert" pod "220a9b39-e990-44ed-bdf7-4c039fa4a9ec" (UID: "220a9b39-e990-44ed-bdf7-4c039fa4a9ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:58.311748 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.311724    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-socket" pod "220a9b39-e990-44ed-bdf7-4c039fa4a9ec" (UID: "220a9b39-e990-44ed-bdf7-4c039fa4a9ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312120    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312147    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c2db3f-6464-47e8-af64-2c432474e832-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312177    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312204    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312388    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312415    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312431    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312445    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312467    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c2db3f-6464-47e8-af64-2c432474e832-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312481    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312495    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312509    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312523    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312539    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312554    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312569    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312583    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312599    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c312869-83d6-406b-bff4-6dc3068df5bf-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312617    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312631    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5071cf6-026d-407b-84b3-f682c15a8d22-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312649    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h977d\" (UniqueName: \"kubernetes.io/projected/11c2db3f-6464-47e8-af64-2c432474e832-kube-api-access-h977d\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.312696 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.312664    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjt2p\" (UniqueName: \"kubernetes.io/projected/0c312869-83d6-406b-bff4-6dc3068df5bf-kube-api-access-vjt2p\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.314008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.313063    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-envoy" pod "f5071cf6-026d-407b-84b3-f682c15a8d22" (UID: "f5071cf6-026d-407b-84b3-f682c15a8d22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.314008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.313135    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-podinfo" pod "f5071cf6-026d-407b-84b3-f682c15a8d22" (UID: "f5071cf6-026d-407b-84b3-f682c15a8d22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:58.314008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.313298    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-kube-api-access-cg88d" pod "f5071cf6-026d-407b-84b3-f682c15a8d22" (UID: "f5071cf6-026d-407b-84b3-f682c15a8d22"). InnerVolumeSpecName "kube-api-access-cg88d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.314008 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.313420    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-token" pod "f5071cf6-026d-407b-84b3-f682c15a8d22" (UID: "f5071cf6-026d-407b-84b3-f682c15a8d22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.314160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.314003    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-envoy" pod "220a9b39-e990-44ed-bdf7-4c039fa4a9ec" (UID: "220a9b39-e990-44ed-bdf7-4c039fa4a9ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.314160 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.314105    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-podinfo" pod "220a9b39-e990-44ed-bdf7-4c039fa4a9ec" (UID: "220a9b39-e990-44ed-bdf7-4c039fa4a9ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:58.314327 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.314307    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-token" pod "220a9b39-e990-44ed-bdf7-4c039fa4a9ec" (UID: "220a9b39-e990-44ed-bdf7-4c039fa4a9ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.314372 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.314332    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-kube-api-access-prn5v" pod "220a9b39-e990-44ed-bdf7-4c039fa4a9ec" (UID: "220a9b39-e990-44ed-bdf7-4c039fa4a9ec"). InnerVolumeSpecName "kube-api-access-prn5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.414091 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.414054    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.414091 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.414084    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.414091 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.414096    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg88d\" (UniqueName: \"kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-kube-api-access-cg88d\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.414548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.414106    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.414548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.414115    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prn5v\" (UniqueName: \"kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-kube-api-access-prn5v\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.414548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.414124    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5071cf6-026d-407b-84b3-f682c15a8d22-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.414548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.414132    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.414548 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.414140    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/220a9b39-e990-44ed-bdf7-4c039fa4a9ec-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.621347 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.621321    4449 generic.go:356] "Generic (PLEG): container finished" podID="220a9b39-e990-44ed-bdf7-4c039fa4a9ec" containerID="dc48825f6451ccdc587ee88c6d5c207a70a0b4deefadf8dd3566e45e84f40832" exitCode=0
Nov 02 09:58:58.621518 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.621365    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"
Nov 02 09:58:58.621518 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.621415    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f" event={"ID":"220a9b39-e990-44ed-bdf7-4c039fa4a9ec","Type":"ContainerDied","Data":"dc48825f6451ccdc587ee88c6d5c207a70a0b4deefadf8dd3566e45e84f40832"}
Nov 02 09:58:58.621518 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.621468    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f" event={"ID":"220a9b39-e990-44ed-bdf7-4c039fa4a9ec","Type":"ContainerDied","Data":"db1c0083c394d9c77d17aee1e0cf779aab8820a7f40d10bf9de703b6c78fdcc6"}
Nov 02 09:58:58.621518 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.621490    4449 scope.go:122] "RemoveContainer" containerID="dc48825f6451ccdc587ee88c6d5c207a70a0b4deefadf8dd3566e45e84f40832"
Nov 02 09:58:58.623107 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.623066    4449 generic.go:356] "Generic (PLEG): container finished" podID="11c2db3f-6464-47e8-af64-2c432474e832" containerID="20bc5977a89804e49285c5d35b26f3dd0ff7a62db9534c2336e1c4914add7854" exitCode=0
Nov 02 09:58:58.623211 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.623109    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"
Nov 02 09:58:58.623211 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.623137    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" event={"ID":"11c2db3f-6464-47e8-af64-2c432474e832","Type":"ContainerDied","Data":"20bc5977a89804e49285c5d35b26f3dd0ff7a62db9534c2336e1c4914add7854"}
Nov 02 09:58:58.623211 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.623182    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns" event={"ID":"11c2db3f-6464-47e8-af64-2c432474e832","Type":"ContainerDied","Data":"7df62c34e349721a5314dabfbc0c20b54d24d1a6d2f6e289c8f8488ebd9eb09b"}
Nov 02 09:58:58.624857 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.624831    4449 generic.go:356] "Generic (PLEG): container finished" podID="f5071cf6-026d-407b-84b3-f682c15a8d22" containerID="19c2cba41a689ece661d978de644dc68f5e8763df72fb8f2e0381df12911d14d" exitCode=0
Nov 02 09:58:58.624965 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.624933    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"
Nov 02 09:58:58.625019 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.624966    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm" event={"ID":"f5071cf6-026d-407b-84b3-f682c15a8d22","Type":"ContainerDied","Data":"19c2cba41a689ece661d978de644dc68f5e8763df72fb8f2e0381df12911d14d"}
Nov 02 09:58:58.625019 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.625008    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm" event={"ID":"f5071cf6-026d-407b-84b3-f682c15a8d22","Type":"ContainerDied","Data":"53ab7a7d8875b4e11fb7598fb7d94806455a4be6b60eb0883563b2440e4ca6da"}
Nov 02 09:58:58.628809 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.628782    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd" event={"ID":"7701534c-2a3d-4211-a06a-2af4a6b454f7","Type":"ContainerStarted","Data":"85841e3b072864c9d711b1b4d0a1034e3a20465340e1e228e02cb73ccf638afd"}
Nov 02 09:58:58.628916 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.628830    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd" event={"ID":"7701534c-2a3d-4211-a06a-2af4a6b454f7","Type":"ContainerStarted","Data":"ee89834e732c50b4e3534882fa7fa08aff99ecd9006e2bc32cf4b7ab758f432f"}
Nov 02 09:58:58.631512 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.631474    4449 generic.go:356] "Generic (PLEG): container finished" podID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerID="2a00e9bcc65ad9494ec7b859ae35cd64b37b4985c196c0e9f6857d02d9fd5e86" exitCode=0
Nov 02 09:58:58.631592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.631523    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"
Nov 02 09:58:58.631592 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.631557    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" event={"ID":"0c312869-83d6-406b-bff4-6dc3068df5bf","Type":"ContainerDied","Data":"2a00e9bcc65ad9494ec7b859ae35cd64b37b4985c196c0e9f6857d02d9fd5e86"}
Nov 02 09:58:58.631714 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.631598    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg" event={"ID":"0c312869-83d6-406b-bff4-6dc3068df5bf","Type":"ContainerDied","Data":"21b9bdb288c6f7f9d516f8a3b68c72e2a7df7cfac5e8599261cf6e10e16afdaf"}
Nov 02 09:58:58.635098 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.635077    4449 scope.go:122] "RemoveContainer" containerID="dc48825f6451ccdc587ee88c6d5c207a70a0b4deefadf8dd3566e45e84f40832"
Nov 02 09:58:58.635415 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:58.635398    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc48825f6451ccdc587ee88c6d5c207a70a0b4deefadf8dd3566e45e84f40832\": container with ID starting with dc48825f6451ccdc587ee88c6d5c207a70a0b4deefadf8dd3566e45e84f40832 not found: ID does not exist" containerID="dc48825f6451ccdc587ee88c6d5c207a70a0b4deefadf8dd3566e45e84f40832"
Nov 02 09:58:58.635474 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.635423    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc48825f6451ccdc587ee88c6d5c207a70a0b4deefadf8dd3566e45e84f40832"} err="failed to get container status \"dc48825f6451ccdc587ee88c6d5c207a70a0b4deefadf8dd3566e45e84f40832\": rpc error: code = NotFound desc = could not find container \"dc48825f6451ccdc587ee88c6d5c207a70a0b4deefadf8dd3566e45e84f40832\": container with ID starting with dc48825f6451ccdc587ee88c6d5c207a70a0b4deefadf8dd3566e45e84f40832 not found: ID does not exist"
Nov 02 09:58:58.635474 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.635441    4449 scope.go:122] "RemoveContainer" containerID="20bc5977a89804e49285c5d35b26f3dd0ff7a62db9534c2336e1c4914add7854"
Nov 02 09:58:58.646882 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.646865    4449 scope.go:122] "RemoveContainer" containerID="20bc5977a89804e49285c5d35b26f3dd0ff7a62db9534c2336e1c4914add7854"
Nov 02 09:58:58.647147 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:58.647126    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20bc5977a89804e49285c5d35b26f3dd0ff7a62db9534c2336e1c4914add7854\": container with ID starting with 20bc5977a89804e49285c5d35b26f3dd0ff7a62db9534c2336e1c4914add7854 not found: ID does not exist" containerID="20bc5977a89804e49285c5d35b26f3dd0ff7a62db9534c2336e1c4914add7854"
Nov 02 09:58:58.647208 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.647154    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20bc5977a89804e49285c5d35b26f3dd0ff7a62db9534c2336e1c4914add7854"} err="failed to get container status \"20bc5977a89804e49285c5d35b26f3dd0ff7a62db9534c2336e1c4914add7854\": rpc error: code = NotFound desc = could not find container \"20bc5977a89804e49285c5d35b26f3dd0ff7a62db9534c2336e1c4914add7854\": container with ID starting with 20bc5977a89804e49285c5d35b26f3dd0ff7a62db9534c2336e1c4914add7854 not found: ID does not exist"
Nov 02 09:58:58.647208 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.647174    4449 scope.go:122] "RemoveContainer" containerID="19c2cba41a689ece661d978de644dc68f5e8763df72fb8f2e0381df12911d14d"
Nov 02 09:58:58.654663 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.654623    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd" podStartSLOduration=1.654610701 podStartE2EDuration="1.654610701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:58:58.652602386 +0000 UTC m=+1415.590889182" watchObservedRunningTime="2026-11-02 09:58:58.654610701 +0000 UTC m=+1415.592897570"
Nov 02 09:58:58.668685 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.668667    4449 scope.go:122] "RemoveContainer" containerID="19c2cba41a689ece661d978de644dc68f5e8763df72fb8f2e0381df12911d14d"
Nov 02 09:58:58.668995 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:58.668976    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c2cba41a689ece661d978de644dc68f5e8763df72fb8f2e0381df12911d14d\": container with ID starting with 19c2cba41a689ece661d978de644dc68f5e8763df72fb8f2e0381df12911d14d not found: ID does not exist" containerID="19c2cba41a689ece661d978de644dc68f5e8763df72fb8f2e0381df12911d14d"
Nov 02 09:58:58.669060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.669001    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c2cba41a689ece661d978de644dc68f5e8763df72fb8f2e0381df12911d14d"} err="failed to get container status \"19c2cba41a689ece661d978de644dc68f5e8763df72fb8f2e0381df12911d14d\": rpc error: code = NotFound desc = could not find container \"19c2cba41a689ece661d978de644dc68f5e8763df72fb8f2e0381df12911d14d\": container with ID starting with 19c2cba41a689ece661d978de644dc68f5e8763df72fb8f2e0381df12911d14d not found: ID does not exist"
Nov 02 09:58:58.669060 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.669018    4449 scope.go:122] "RemoveContainer" containerID="2a00e9bcc65ad9494ec7b859ae35cd64b37b4985c196c0e9f6857d02d9fd5e86"
Nov 02 09:58:58.676914 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.676889    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"]
Nov 02 09:58:58.679550 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.679432    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-pgrhg"]
Nov 02 09:58:58.690136 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.690117    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"]
Nov 02 09:58:58.696699 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.696681    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-rgcns"]
Nov 02 09:58:58.709506 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.709480    4449 scope.go:122] "RemoveContainer" containerID="fc47298d4dbb14070fd07471ea1613b082cf73f8b47a7ced1dbf45924739a10d"
Nov 02 09:58:58.711095 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.711078    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"]
Nov 02 09:58:58.716901 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.716854    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-jzm5f"]
Nov 02 09:58:58.724140 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.724120    4449 scope.go:122] "RemoveContainer" containerID="2a00e9bcc65ad9494ec7b859ae35cd64b37b4985c196c0e9f6857d02d9fd5e86"
Nov 02 09:58:58.724667 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:58.724648    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a00e9bcc65ad9494ec7b859ae35cd64b37b4985c196c0e9f6857d02d9fd5e86\": container with ID starting with 2a00e9bcc65ad9494ec7b859ae35cd64b37b4985c196c0e9f6857d02d9fd5e86 not found: ID does not exist" containerID="2a00e9bcc65ad9494ec7b859ae35cd64b37b4985c196c0e9f6857d02d9fd5e86"
Nov 02 09:58:58.724744 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.724677    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a00e9bcc65ad9494ec7b859ae35cd64b37b4985c196c0e9f6857d02d9fd5e86"} err="failed to get container status \"2a00e9bcc65ad9494ec7b859ae35cd64b37b4985c196c0e9f6857d02d9fd5e86\": rpc error: code = NotFound desc = could not find container \"2a00e9bcc65ad9494ec7b859ae35cd64b37b4985c196c0e9f6857d02d9fd5e86\": container with ID starting with 2a00e9bcc65ad9494ec7b859ae35cd64b37b4985c196c0e9f6857d02d9fd5e86 not found: ID does not exist"
Nov 02 09:58:58.724744 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.724694    4449 scope.go:122] "RemoveContainer" containerID="fc47298d4dbb14070fd07471ea1613b082cf73f8b47a7ced1dbf45924739a10d"
Nov 02 09:58:58.725009 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:58:58.724988    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc47298d4dbb14070fd07471ea1613b082cf73f8b47a7ced1dbf45924739a10d\": container with ID starting with fc47298d4dbb14070fd07471ea1613b082cf73f8b47a7ced1dbf45924739a10d not found: ID does not exist" containerID="fc47298d4dbb14070fd07471ea1613b082cf73f8b47a7ced1dbf45924739a10d"
Nov 02 09:58:58.725116 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.725014    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc47298d4dbb14070fd07471ea1613b082cf73f8b47a7ced1dbf45924739a10d"} err="failed to get container status \"fc47298d4dbb14070fd07471ea1613b082cf73f8b47a7ced1dbf45924739a10d\": rpc error: code = NotFound desc = could not find container \"fc47298d4dbb14070fd07471ea1613b082cf73f8b47a7ced1dbf45924739a10d\": container with ID starting with fc47298d4dbb14070fd07471ea1613b082cf73f8b47a7ced1dbf45924739a10d not found: ID does not exist"
Nov 02 09:58:58.731892 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.731869    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"]
Nov 02 09:58:58.735199 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:58.735183    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-6mdlm"]
Nov 02 09:58:59.113494 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:59.113457    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds" podUID="b57143d3-3a7a-4aa0-84e5-09e0b155a963" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:58:59.207784 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:59.207750    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" path="/var/lib/kubelet/pods/0c312869-83d6-406b-bff4-6dc3068df5bf/volumes"
Nov 02 09:58:59.208280 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:59.208265    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c2db3f-6464-47e8-af64-2c432474e832" path="/var/lib/kubelet/pods/11c2db3f-6464-47e8-af64-2c432474e832/volumes"
Nov 02 09:58:59.208726 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:59.208712    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220a9b39-e990-44ed-bdf7-4c039fa4a9ec" path="/var/lib/kubelet/pods/220a9b39-e990-44ed-bdf7-4c039fa4a9ec/volumes"
Nov 02 09:58:59.209563 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:59.209545    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5071cf6-026d-407b-84b3-f682c15a8d22" path="/var/lib/kubelet/pods/f5071cf6-026d-407b-84b3-f682c15a8d22/volumes"
Nov 02 09:58:59.643409 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:59.643375    4449 generic.go:356] "Generic (PLEG): container finished" podID="7701534c-2a3d-4211-a06a-2af4a6b454f7" containerID="85841e3b072864c9d711b1b4d0a1034e3a20465340e1e228e02cb73ccf638afd" exitCode=0
Nov 02 09:58:59.643832 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:58:59.643462    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd" event={"ID":"7701534c-2a3d-4211-a06a-2af4a6b454f7","Type":"ContainerDied","Data":"85841e3b072864c9d711b1b4d0a1034e3a20465340e1e228e02cb73ccf638afd"}
Nov 02 09:59:00.806013 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:00.805987    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"
Nov 02 09:59:00.846777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:00.846746    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7701534c-2a3d-4211-a06a-2af4a6b454f7-kube-api-access-zvnbz\" (UniqueName: \"kubernetes.io/projected/7701534c-2a3d-4211-a06a-2af4a6b454f7-kube-api-access-zvnbz\") pod \"7701534c-2a3d-4211-a06a-2af4a6b454f7\" (UID: \"7701534c-2a3d-4211-a06a-2af4a6b454f7\") "
Nov 02 09:59:00.846981 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:00.846801    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-openshift-ca\") pod \"7701534c-2a3d-4211-a06a-2af4a6b454f7\" (UID: \"7701534c-2a3d-4211-a06a-2af4a6b454f7\") "
Nov 02 09:59:00.846981 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:00.846954    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-xks-ca\") pod \"7701534c-2a3d-4211-a06a-2af4a6b454f7\" (UID: \"7701534c-2a3d-4211-a06a-2af4a6b454f7\") "
Nov 02 09:59:00.847263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:00.847236    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-openshift-ca" pod "7701534c-2a3d-4211-a06a-2af4a6b454f7" (UID: "7701534c-2a3d-4211-a06a-2af4a6b454f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:59:00.847483 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:00.847458    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:59:00.849291 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:00.849258    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7701534c-2a3d-4211-a06a-2af4a6b454f7-kube-api-access-zvnbz" pod "7701534c-2a3d-4211-a06a-2af4a6b454f7" (UID: "7701534c-2a3d-4211-a06a-2af4a6b454f7"). InnerVolumeSpecName "kube-api-access-zvnbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:59:00.849543 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:00.849516    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-xks-ca" pod "7701534c-2a3d-4211-a06a-2af4a6b454f7" (UID: "7701534c-2a3d-4211-a06a-2af4a6b454f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:59:00.949170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:00.949075    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvnbz\" (UniqueName: \"kubernetes.io/projected/7701534c-2a3d-4211-a06a-2af4a6b454f7-kube-api-access-zvnbz\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:59:00.949170 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:00.949116    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7701534c-2a3d-4211-a06a-2af4a6b454f7-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:59:01.669383 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:01.669354    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"
Nov 02 09:59:01.669621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:01.669389    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd" event={"ID":"7701534c-2a3d-4211-a06a-2af4a6b454f7","Type":"ContainerDied","Data":"ee89834e732c50b4e3534882fa7fa08aff99ecd9006e2bc32cf4b7ab758f432f"}
Nov 02 09:59:01.669621 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:01.669440    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee89834e732c50b4e3534882fa7fa08aff99ecd9006e2bc32cf4b7ab758f432f"
Nov 02 09:59:06.027841 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:06.027794    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"]
Nov 02 09:59:06.035804 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:06.035768    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1fa4ab-dl5mf"]
Nov 02 09:59:06.547910 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:59:06.547878    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:59:07.206581 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.206536    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44bc7caa-a813-480b-adb7-e7a41fb91d99" path="/var/lib/kubelet/pods/44bc7caa-a813-480b-adb7-e7a41fb91d99/volumes"
Nov 02 09:59:07.705575 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.705547    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"
Nov 02 09:59:07.737907 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.737870    4449 generic.go:356] "Generic (PLEG): container finished" podID="b57143d3-3a7a-4aa0-84e5-09e0b155a963" containerID="41b7b48494329bf7d1e9b3a666a856a1077cf0876c94bb6064d071d0a77781a1" exitCode=0
Nov 02 09:59:07.738080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.737933    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"
Nov 02 09:59:07.738080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.737964    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds" event={"ID":"b57143d3-3a7a-4aa0-84e5-09e0b155a963","Type":"ContainerDied","Data":"41b7b48494329bf7d1e9b3a666a856a1077cf0876c94bb6064d071d0a77781a1"}
Nov 02 09:59:07.738080 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.738062    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds" event={"ID":"b57143d3-3a7a-4aa0-84e5-09e0b155a963","Type":"ContainerDied","Data":"8a2d76c25984d0d4bd4136e4c45cbb28f965b594ea37ef9b9786956b9a8979ef"}
Nov 02 09:59:07.738267 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.738093    4449 scope.go:122] "RemoveContainer" containerID="41b7b48494329bf7d1e9b3a666a856a1077cf0876c94bb6064d071d0a77781a1"
Nov 02 09:59:07.740517 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.740496    4449 generic.go:356] "Generic (PLEG): container finished" podID="c419bca8-6a23-4b7d-bfa1-c980a92c2e08" containerID="8f459152acbd27aea887ab3a5637230d25782dd6051a6c5ca6b80ba7d8f09a0f" exitCode=0
Nov 02 09:59:07.740637 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.740569    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp" event={"ID":"c419bca8-6a23-4b7d-bfa1-c980a92c2e08","Type":"ContainerDied","Data":"8f459152acbd27aea887ab3a5637230d25782dd6051a6c5ca6b80ba7d8f09a0f"}
Nov 02 09:59:07.752575 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.752557    4449 scope.go:122] "RemoveContainer" containerID="41b7b48494329bf7d1e9b3a666a856a1077cf0876c94bb6064d071d0a77781a1"
Nov 02 09:59:07.752897 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:59:07.752872    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b7b48494329bf7d1e9b3a666a856a1077cf0876c94bb6064d071d0a77781a1\": container with ID starting with 41b7b48494329bf7d1e9b3a666a856a1077cf0876c94bb6064d071d0a77781a1 not found: ID does not exist" containerID="41b7b48494329bf7d1e9b3a666a856a1077cf0876c94bb6064d071d0a77781a1"
Nov 02 09:59:07.752955 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.752913    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b7b48494329bf7d1e9b3a666a856a1077cf0876c94bb6064d071d0a77781a1"} err="failed to get container status \"41b7b48494329bf7d1e9b3a666a856a1077cf0876c94bb6064d071d0a77781a1\": rpc error: code = NotFound desc = could not find container \"41b7b48494329bf7d1e9b3a666a856a1077cf0876c94bb6064d071d0a77781a1\": container with ID starting with 41b7b48494329bf7d1e9b3a666a856a1077cf0876c94bb6064d071d0a77781a1 not found: ID does not exist"
Nov 02 09:59:07.833177 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.833155    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"
Nov 02 09:59:07.836532 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.836509    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b57143d3-3a7a-4aa0-84e5-09e0b155a963-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57143d3-3a7a-4aa0-84e5-09e0b155a963-plugins-config-volume\") pod \"b57143d3-3a7a-4aa0-84e5-09e0b155a963\" (UID: \"b57143d3-3a7a-4aa0-84e5-09e0b155a963\") "
Nov 02 09:59:07.836705 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.836687    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57143d3-3a7a-4aa0-84e5-09e0b155a963-kube-api-access-chrtq\" (UniqueName: \"kubernetes.io/projected/b57143d3-3a7a-4aa0-84e5-09e0b155a963-kube-api-access-chrtq\") pod \"b57143d3-3a7a-4aa0-84e5-09e0b155a963\" (UID: \"b57143d3-3a7a-4aa0-84e5-09e0b155a963\") "
Nov 02 09:59:07.836991 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.836967    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57143d3-3a7a-4aa0-84e5-09e0b155a963-plugins-config-volume" pod "b57143d3-3a7a-4aa0-84e5-09e0b155a963" (UID: "b57143d3-3a7a-4aa0-84e5-09e0b155a963"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:59:07.837164 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.837144    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57143d3-3a7a-4aa0-84e5-09e0b155a963-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:59:07.838959 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.838937    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57143d3-3a7a-4aa0-84e5-09e0b155a963-kube-api-access-chrtq" pod "b57143d3-3a7a-4aa0-84e5-09e0b155a963" (UID: "b57143d3-3a7a-4aa0-84e5-09e0b155a963"). InnerVolumeSpecName "kube-api-access-chrtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:59:07.938014 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.937985    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-kube-api-access-87gbm\" (UniqueName: \"kubernetes.io/projected/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-kube-api-access-87gbm\") pod \"c419bca8-6a23-4b7d-bfa1-c980a92c2e08\" (UID: \"c419bca8-6a23-4b7d-bfa1-c980a92c2e08\") "
Nov 02 09:59:07.938263 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.938031    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-plugins-config-volume\") pod \"c419bca8-6a23-4b7d-bfa1-c980a92c2e08\" (UID: \"c419bca8-6a23-4b7d-bfa1-c980a92c2e08\") "
Nov 02 09:59:07.938540 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.938511    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-plugins-config-volume" pod "c419bca8-6a23-4b7d-bfa1-c980a92c2e08" (UID: "c419bca8-6a23-4b7d-bfa1-c980a92c2e08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:59:07.938643 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.938593    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chrtq\" (UniqueName: \"kubernetes.io/projected/b57143d3-3a7a-4aa0-84e5-09e0b155a963-kube-api-access-chrtq\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:59:07.940266 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:07.940247    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-kube-api-access-87gbm" pod "c419bca8-6a23-4b7d-bfa1-c980a92c2e08" (UID: "c419bca8-6a23-4b7d-bfa1-c980a92c2e08"). InnerVolumeSpecName "kube-api-access-87gbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:59:08.039253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:08.039201    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87gbm\" (UniqueName: \"kubernetes.io/projected/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-kube-api-access-87gbm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:59:08.039253 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:08.039247    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c419bca8-6a23-4b7d-bfa1-c980a92c2e08-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:59:08.063862 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:08.063789    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"]
Nov 02 09:59:08.069611 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:08.069585    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9e7156-f747b4965-v9sds"]
Nov 02 09:59:08.751778 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:08.751750    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"
Nov 02 09:59:08.752256 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:08.751755    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp" event={"ID":"c419bca8-6a23-4b7d-bfa1-c980a92c2e08","Type":"ContainerDied","Data":"498847fcc56c8efa778c492b6e7d71571f495761ab8ff2179c72e2e7a7e7e5a8"}
Nov 02 09:59:08.752256 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:08.751871    4449 scope.go:122] "RemoveContainer" containerID="8f459152acbd27aea887ab3a5637230d25782dd6051a6c5ca6b80ba7d8f09a0f"
Nov 02 09:59:08.777604 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:08.777571    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"]
Nov 02 09:59:08.783762 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:08.783740    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-9e7156-7957df6fc5-ctbhp"]
Nov 02 09:59:09.206965 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:09.206933    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b57143d3-3a7a-4aa0-84e5-09e0b155a963" path="/var/lib/kubelet/pods/b57143d3-3a7a-4aa0-84e5-09e0b155a963/volumes"
Nov 02 09:59:09.207386 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:09.207371    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c419bca8-6a23-4b7d-bfa1-c980a92c2e08" path="/var/lib/kubelet/pods/c419bca8-6a23-4b7d-bfa1-c980a92c2e08/volumes"
Nov 02 09:59:10.360379 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.360345    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"]
Nov 02 09:59:10.360911 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.360897    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f5071cf6-026d-407b-84b3-f682c15a8d22" containerName="istio-proxy"
Nov 02 09:59:10.360957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.360912    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5071cf6-026d-407b-84b3-f682c15a8d22" containerName="istio-proxy"
Nov 02 09:59:10.360957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.360923    4449 cpu_manager.go:414] "containerMap: removing container" podUID="c419bca8-6a23-4b7d-bfa1-c980a92c2e08" containerName="payload-pre-processing"
Nov 02 09:59:10.360957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.360928    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c419bca8-6a23-4b7d-bfa1-c980a92c2e08" containerName="payload-pre-processing"
Nov 02 09:59:10.360957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.360937    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy"
Nov 02 09:59:10.360957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.360942    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy"
Nov 02 09:59:10.360957 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.360956    4449 cpu_manager.go:414] "containerMap: removing container" podUID="7701534c-2a3d-4211-a06a-2af4a6b454f7" containerName="revoke-keys"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.360962    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7701534c-2a3d-4211-a06a-2af4a6b454f7" containerName="revoke-keys"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.360972    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b57143d3-3a7a-4aa0-84e5-09e0b155a963" containerName="payload-processing"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.360976    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57143d3-3a7a-4aa0-84e5-09e0b155a963" containerName="payload-processing"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.360991    4449 cpu_manager.go:414] "containerMap: removing container" podUID="220a9b39-e990-44ed-bdf7-4c039fa4a9ec" containerName="istio-proxy"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.360995    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220a9b39-e990-44ed-bdf7-4c039fa4a9ec" containerName="istio-proxy"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.361006    4449 cpu_manager.go:414] "containerMap: removing container" podUID="11c2db3f-6464-47e8-af64-2c432474e832" containerName="istio-proxy"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.361010    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c2db3f-6464-47e8-af64-2c432474e832" containerName="istio-proxy"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.361087    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="220a9b39-e990-44ed-bdf7-4c039fa4a9ec" containerName="istio-proxy"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.361095    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c2db3f-6464-47e8-af64-2c432474e832" containerName="istio-proxy"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.361103    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5071cf6-026d-407b-84b3-f682c15a8d22" containerName="istio-proxy"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.361110    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="c419bca8-6a23-4b7d-bfa1-c980a92c2e08" containerName="payload-pre-processing"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.361116    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.361121    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.361128    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="7701534c-2a3d-4211-a06a-2af4a6b454f7" containerName="revoke-keys"
Nov 02 09:59:10.361190 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.361134    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57143d3-3a7a-4aa0-84e5-09e0b155a963" containerName="payload-processing"
Nov 02 09:59:10.366002 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.365984    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"
Nov 02 09:59:10.370655 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.370125    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:59:10.370655 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.370357    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:59:10.370863 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.370829    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:59:10.371088 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.371072    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:59:10.373981 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.373947    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"]
Nov 02 09:59:10.468417 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.468386    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39cc5919-fhb2t\" (UID: \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"
Nov 02 09:59:10.468612 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.468526    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-796ql\" (UniqueName: \"kubernetes.io/projected/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-kube-api-access-796ql\") pod \"maas-api-revoke-sub-39cc5919-fhb2t\" (UID: \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"
Nov 02 09:59:10.468612 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.468578    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39cc5919-fhb2t\" (UID: \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"
Nov 02 09:59:10.569989 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.569952    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39cc5919-fhb2t\" (UID: \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"
Nov 02 09:59:10.570176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.570051    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-796ql\" (UniqueName: \"kubernetes.io/projected/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-kube-api-access-796ql\") pod \"maas-api-revoke-sub-39cc5919-fhb2t\" (UID: \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"
Nov 02 09:59:10.570176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.570084    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39cc5919-fhb2t\" (UID: \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"
Nov 02 09:59:10.570777 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.570746    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39cc5919-fhb2t\" (UID: \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"
Nov 02 09:59:10.572695 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.572675    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39cc5919-fhb2t\" (UID: \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"
Nov 02 09:59:10.583169 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.583145    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-796ql\" (UniqueName: \"kubernetes.io/projected/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-kube-api-access-796ql\") pod \"maas-api-revoke-sub-39cc5919-fhb2t\" (UID: \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"
Nov 02 09:59:10.683151 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.683067    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"
Nov 02 09:59:10.976362 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:59:10.976333    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode107f26f_04b1_44e5_ad54_1e42fc4c09ea.slice/crio-54edd475eed0c1a26c21b307c1057e18034cd05fc0afe400f9710ca3aab46b0c WatchSource:0}: Error finding container 54edd475eed0c1a26c21b307c1057e18034cd05fc0afe400f9710ca3aab46b0c: Status 404 returned error can't find the container with id 54edd475eed0c1a26c21b307c1057e18034cd05fc0afe400f9710ca3aab46b0c
Nov 02 09:59:10.976644 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:10.976617    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"]
Nov 02 09:59:11.783345 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:11.783310    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t" event={"ID":"e107f26f-04b1-44e5-ad54-1e42fc4c09ea","Type":"ContainerStarted","Data":"5e41a3a1ce260d51a717aab97bf47a5c55685c329a42b23c315ce626e6fb9396"}
Nov 02 09:59:11.783750 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:11.783370    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t" event={"ID":"e107f26f-04b1-44e5-ad54-1e42fc4c09ea","Type":"ContainerStarted","Data":"54edd475eed0c1a26c21b307c1057e18034cd05fc0afe400f9710ca3aab46b0c"}
Nov 02 09:59:11.807555 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:11.807500    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t" podStartSLOduration=1.8074860240000001 podStartE2EDuration="1.807486024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:59:11.804579242 +0000 UTC m=+1428.742866045" watchObservedRunningTime="2026-11-02 09:59:11.807486024 +0000 UTC m=+1428.745772799"
Nov 02 09:59:12.793901 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:12.793864    4449 generic.go:356] "Generic (PLEG): container finished" podID="e107f26f-04b1-44e5-ad54-1e42fc4c09ea" containerID="5e41a3a1ce260d51a717aab97bf47a5c55685c329a42b23c315ce626e6fb9396" exitCode=0
Nov 02 09:59:12.794382 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:12.793948    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t" event={"ID":"e107f26f-04b1-44e5-ad54-1e42fc4c09ea","Type":"ContainerDied","Data":"5e41a3a1ce260d51a717aab97bf47a5c55685c329a42b23c315ce626e6fb9396"}
Nov 02 09:59:13.962516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:13.962488    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"
Nov 02 09:59:14.115001 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:14.114905    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-xks-ca\") pod \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\" (UID: \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\") "
Nov 02 09:59:14.115194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:14.115040    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-kube-api-access-796ql\" (UniqueName: \"kubernetes.io/projected/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-kube-api-access-796ql\") pod \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\" (UID: \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\") "
Nov 02 09:59:14.115194 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:14.115101    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-openshift-ca\") pod \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\" (UID: \"e107f26f-04b1-44e5-ad54-1e42fc4c09ea\") "
Nov 02 09:59:14.115521 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:14.115490    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-openshift-ca" pod "e107f26f-04b1-44e5-ad54-1e42fc4c09ea" (UID: "e107f26f-04b1-44e5-ad54-1e42fc4c09ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:59:14.115628 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:14.115606    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:59:14.117368 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:14.117345    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-xks-ca" pod "e107f26f-04b1-44e5-ad54-1e42fc4c09ea" (UID: "e107f26f-04b1-44e5-ad54-1e42fc4c09ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:59:14.117439 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:14.117377    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-kube-api-access-796ql" pod "e107f26f-04b1-44e5-ad54-1e42fc4c09ea" (UID: "e107f26f-04b1-44e5-ad54-1e42fc4c09ea"). InnerVolumeSpecName "kube-api-access-796ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:59:14.216681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:14.216641    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-796ql\" (UniqueName: \"kubernetes.io/projected/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-kube-api-access-796ql\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:59:14.216681 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:14.216675    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e107f26f-04b1-44e5-ad54-1e42fc4c09ea-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:59:14.814647 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:14.814602    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t" event={"ID":"e107f26f-04b1-44e5-ad54-1e42fc4c09ea","Type":"ContainerDied","Data":"54edd475eed0c1a26c21b307c1057e18034cd05fc0afe400f9710ca3aab46b0c"}
Nov 02 09:59:14.814870 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:14.814643    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"
Nov 02 09:59:14.814870 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:14.814705    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54edd475eed0c1a26c21b307c1057e18034cd05fc0afe400f9710ca3aab46b0c"
Nov 02 09:59:16.026745 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:16.026709    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"]
Nov 02 09:59:16.031499 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:16.031469    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51d514f9-jmlmm"]
Nov 02 09:59:17.214269 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:17.213978    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72589ec-4fb0-4348-a63f-57178b3623e3" path="/var/lib/kubelet/pods/b72589ec-4fb0-4348-a63f-57178b3623e3/volumes"
Nov 02 09:59:23.025430 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:23.025391    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"]
Nov 02 09:59:23.028973 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:23.028945    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2453a69-qqxdl"]
Nov 02 09:59:23.206684 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:23.206651    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e675a17-aed8-48b0-96c9-d9ccfd5c3121" path="/var/lib/kubelet/pods/7e675a17-aed8-48b0-96c9-d9ccfd5c3121/volumes"
Nov 02 09:59:25.230780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:25.230744    4449 scope.go:122] "RemoveContainer" containerID="71e063efbf0e65e960223db68d239dafb8482c6624e0153cd194b901b34df338"
Nov 02 09:59:25.241433 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:25.241410    4449 scope.go:122] "RemoveContainer" containerID="f8384f4e75e8278bffb92a4d9484817dc9f631af634b9cf46b890562875f8fa0"
Nov 02 09:59:25.251402 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:25.251380    4449 scope.go:122] "RemoveContainer" containerID="0e927a52694b31b7217598e908d5c9c1855983a7d96bd4ecf34ef0e2bc8b8a10"
Nov 02 09:59:25.260660 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:25.260642    4449 scope.go:122] "RemoveContainer" containerID="a2c271eb1e8fcd8fdf1207532627d7da3d304d9c8fa8ba1f1440e99bb019a59f"
Nov 02 09:59:25.270322 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:25.270302    4449 scope.go:122] "RemoveContainer" containerID="a9459e0330b0aef63fba5ed2f75c8e2e6dd220ee5bc81ec51a26f54214ed5446"
Nov 02 09:59:25.279307 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:25.279288    4449 scope.go:122] "RemoveContainer" containerID="65c1a932eb3d1e1e120d8348d3403ea6fba1f7733de9f769752808205d89ba13"
Nov 02 09:59:25.289105 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:25.289082    4449 scope.go:122] "RemoveContainer" containerID="1918b26dcf0c0d6e06a508874438723011103ef58b498f71fe803d51e935335e"
Nov 02 09:59:33.388111 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.388077    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"]
Nov 02 09:59:33.388733 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.388709    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy"
Nov 02 09:59:33.388733 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.388732    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c312869-83d6-406b-bff4-6dc3068df5bf" containerName="istio-proxy"
Nov 02 09:59:33.388945 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.388744    4449 cpu_manager.go:414] "containerMap: removing container" podUID="e107f26f-04b1-44e5-ad54-1e42fc4c09ea" containerName="revoke-keys"
Nov 02 09:59:33.388945 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.388753    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e107f26f-04b1-44e5-ad54-1e42fc4c09ea" containerName="revoke-keys"
Nov 02 09:59:33.388945 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.388891    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="e107f26f-04b1-44e5-ad54-1e42fc4c09ea" containerName="revoke-keys"
Nov 02 09:59:33.394854 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.394827    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.408313 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.408279    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"]
Nov 02 09:59:33.474351 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.474312    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.474531 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.474373    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.474531 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.474448    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95d8r\" (UniqueName: \"kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-kube-api-access-95d8r\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.474531 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.474484    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.474531 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.474506    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.474531 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.474520    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.474712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.474650    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.474712 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.474685    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.474780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.474722    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.575578 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.575536    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.575780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.575589    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.575780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.575615    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95d8r\" (UniqueName: \"kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-kube-api-access-95d8r\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.575780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.575647    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.575780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.575664    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.575780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.575680    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.575780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.575728    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.575780 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.575756    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.576176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.575796    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.576176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.576100    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.576176 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.576155    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.576410 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.576283    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.576410 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.576399    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.576511 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.576445    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.578237 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.578192    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.578544 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.578521    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.588943 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.588916    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.589576 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.589553    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95d8r\" (UniqueName: \"kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-kube-api-access-95d8r\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:33.715509 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:33.715418    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:34.003691 ip-10-0-128-217 kubenswrapper[4449]: W1102 09:59:34.003663    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93519ae_58de_4fe9_bbdf_fd2d607d497c.slice/crio-24f73c22f64a2df17b532ca51e0cff59817a8268da5206c0cd46d8c174d5c573 WatchSource:0}: Error finding container 24f73c22f64a2df17b532ca51e0cff59817a8268da5206c0cd46d8c174d5c573: Status 404 returned error can't find the container with id 24f73c22f64a2df17b532ca51e0cff59817a8268da5206c0cd46d8c174d5c573
Nov 02 09:59:34.005048 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:34.005015    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"]
Nov 02 09:59:34.006179 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:34.006142    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:59:34.006300 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:34.006240    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:59:34.006300 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:34.006284    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 09:59:35.000043 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:35.000006    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j" event={"ID":"f93519ae-58de-4fe9-bbdf-fd2d607d497c","Type":"ContainerStarted","Data":"47e171145fb459186ba8bd726848b362cf44d6c4ba37888fdd4d75677015d215"}
Nov 02 09:59:35.000516 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:35.000093    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j" event={"ID":"f93519ae-58de-4fe9-bbdf-fd2d607d497c","Type":"ContainerStarted","Data":"24f73c22f64a2df17b532ca51e0cff59817a8268da5206c0cd46d8c174d5c573"}
Nov 02 09:59:35.029090 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:35.029033    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j" podStartSLOduration=2.029020917 podStartE2EDuration="2.029020917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:59:35.025441647 +0000 UTC m=+1451.963728442" watchObservedRunningTime="2026-11-02 09:59:35.029020917 +0000 UTC m=+1451.967307693"
Nov 02 09:59:35.716445 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:35.716402    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:35.718138 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:35.718111    4449 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 02 09:59:35.718295 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:35.718160    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j" podUID="f93519ae-58de-4fe9-bbdf-fd2d607d497c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 02 09:59:36.569243 ip-10-0-128-217 kubenswrapper[4449]: E1102 09:59:36.564886    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 09:59:36.716784 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:36.716748    4449 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 02 09:59:36.716944 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:36.716817    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j" podUID="f93519ae-58de-4fe9-bbdf-fd2d607d497c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 02 09:59:37.715916 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:37.715876    4449 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 02 09:59:37.716348 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:37.715944    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j" podUID="f93519ae-58de-4fe9-bbdf-fd2d607d497c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 02 09:59:38.723246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:38.716609    4449 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 02 09:59:38.723246 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:38.716676    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j" podUID="f93519ae-58de-4fe9-bbdf-fd2d607d497c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 02 09:59:39.716744 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:39.716703    4449 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 02 09:59:39.716929 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:39.716763    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j" podUID="f93519ae-58de-4fe9-bbdf-fd2d607d497c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 02 09:59:40.720596 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:40.720564    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:40.721073 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:40.721053    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:40.721844 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:40.721822    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 09:59:59.777633 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:59.777595    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"]
Nov 02 09:59:59.782037 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:59.782012    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"
Nov 02 09:59:59.787687 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:59.787655    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:59:59.787943 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:59.787715    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:59:59.788902 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:59.788663    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:59:59.788902 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:59.788755    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:59:59.791093 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:59.791070    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"]
Nov 02 09:59:59.937829 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:59.937792    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf5xm\" (UniqueName: \"kubernetes.io/projected/b08a49e9-6d49-4ddc-9288-f6a38b620411-kube-api-access-cf5xm\") pod \"maas-api-revoke-sub-91778291-q962v\" (UID: \"b08a49e9-6d49-4ddc-9288-f6a38b620411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"
Nov 02 09:59:59.938016 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:59.937848    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91778291-q962v\" (UID: \"b08a49e9-6d49-4ddc-9288-f6a38b620411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"
Nov 02 09:59:59.938117 ip-10-0-128-217 kubenswrapper[4449]: I1102 09:59:59.938093    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91778291-q962v\" (UID: \"b08a49e9-6d49-4ddc-9288-f6a38b620411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"
Nov 02 10:00:00.039386 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.039351    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91778291-q962v\" (UID: \"b08a49e9-6d49-4ddc-9288-f6a38b620411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"
Nov 02 10:00:00.039595 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.039408    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5xm\" (UniqueName: \"kubernetes.io/projected/b08a49e9-6d49-4ddc-9288-f6a38b620411-kube-api-access-cf5xm\") pod \"maas-api-revoke-sub-91778291-q962v\" (UID: \"b08a49e9-6d49-4ddc-9288-f6a38b620411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"
Nov 02 10:00:00.039595 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.039462    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91778291-q962v\" (UID: \"b08a49e9-6d49-4ddc-9288-f6a38b620411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"
Nov 02 10:00:00.040188 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.040167    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91778291-q962v\" (UID: \"b08a49e9-6d49-4ddc-9288-f6a38b620411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"
Nov 02 10:00:00.041942 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.041919    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91778291-q962v\" (UID: \"b08a49e9-6d49-4ddc-9288-f6a38b620411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"
Nov 02 10:00:00.052566 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.052540    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf5xm\" (UniqueName: \"kubernetes.io/projected/b08a49e9-6d49-4ddc-9288-f6a38b620411-kube-api-access-cf5xm\") pod \"maas-api-revoke-sub-91778291-q962v\" (UID: \"b08a49e9-6d49-4ddc-9288-f6a38b620411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"
Nov 02 10:00:00.102351 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.102308    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"
Nov 02 10:00:00.164713 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.164689    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"]
Nov 02 10:00:00.171808 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.171780    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"
Nov 02 10:00:00.175963 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.175929    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"]
Nov 02 10:00:00.212152 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.212113    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"]
Nov 02 10:00:00.219271 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.213676    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"]
Nov 02 10:00:00.219271 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.214867    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"
Nov 02 10:00:00.342589 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.342476    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893560-tr7jr\" (UID: \"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"
Nov 02 10:00:00.342589 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.342527    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmt58\" (UniqueName: \"kubernetes.io/projected/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-kube-api-access-wmt58\") pod \"maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm\" (UID: \"aa87dab3-ee63-4a13-802f-af3eab2ca8f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"
Nov 02 10:00:00.342844 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.342755    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxjrv\" (UniqueName: \"kubernetes.io/projected/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-kube-api-access-sxjrv\") pod \"maas-api-key-cleanup-29893560-tr7jr\" (UID: \"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"
Nov 02 10:00:00.342963 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.342936    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm\" (UID: \"aa87dab3-ee63-4a13-802f-af3eab2ca8f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"
Nov 02 10:00:00.387124 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:00:00.387092    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08a49e9_6d49_4ddc_9288_f6a38b620411.slice/crio-29e7425ed483ab4b559edb3c15714b705fbd84667ab171ff8f45de56ec192e42 WatchSource:0}: Error finding container 29e7425ed483ab4b559edb3c15714b705fbd84667ab171ff8f45de56ec192e42: Status 404 returned error can't find the container with id 29e7425ed483ab4b559edb3c15714b705fbd84667ab171ff8f45de56ec192e42
Nov 02 10:00:00.387564 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.387527    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"]
Nov 02 10:00:00.444671 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.444631    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxjrv\" (UniqueName: \"kubernetes.io/projected/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-kube-api-access-sxjrv\") pod \"maas-api-key-cleanup-29893560-tr7jr\" (UID: \"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"
Nov 02 10:00:00.444858 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.444735    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm\" (UID: \"aa87dab3-ee63-4a13-802f-af3eab2ca8f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"
Nov 02 10:00:00.444858 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.444815    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893560-tr7jr\" (UID: \"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"
Nov 02 10:00:00.444858 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.444839    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmt58\" (UniqueName: \"kubernetes.io/projected/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-kube-api-access-wmt58\") pod \"maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm\" (UID: \"aa87dab3-ee63-4a13-802f-af3eab2ca8f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"
Nov 02 10:00:00.445505 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.445484    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893560-tr7jr\" (UID: \"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"
Nov 02 10:00:00.445552 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.445486    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm\" (UID: \"aa87dab3-ee63-4a13-802f-af3eab2ca8f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"
Nov 02 10:00:00.462895 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.462862    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxjrv\" (UniqueName: \"kubernetes.io/projected/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-kube-api-access-sxjrv\") pod \"maas-api-key-cleanup-29893560-tr7jr\" (UID: \"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"
Nov 02 10:00:00.463048 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.462989    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmt58\" (UniqueName: \"kubernetes.io/projected/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-kube-api-access-wmt58\") pod \"maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm\" (UID: \"aa87dab3-ee63-4a13-802f-af3eab2ca8f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"
Nov 02 10:00:00.512156 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.512116    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"
Nov 02 10:00:00.538486 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.538450    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"
Nov 02 10:00:00.820471 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:00:00.820432    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa87dab3_ee63_4a13_802f_af3eab2ca8f3.slice/crio-f5f3f180c4448e916d5a8504f325e0d245632effd84f70a9fb0c9edcb1b140b5 WatchSource:0}: Error finding container f5f3f180c4448e916d5a8504f325e0d245632effd84f70a9fb0c9edcb1b140b5: Status 404 returned error can't find the container with id f5f3f180c4448e916d5a8504f325e0d245632effd84f70a9fb0c9edcb1b140b5
Nov 02 10:00:00.825324 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.825296    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"]
Nov 02 10:00:00.865501 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:00.865473    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"]
Nov 02 10:00:00.872561 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:00:00.872534    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0e46e3_ae58_4909_8eaa_e57bbb0a62ec.slice/crio-96424ef1ca09f8ed6dbbeaaecc1efe96887008f6a44d4c7ab69600ab237b648a WatchSource:0}: Error finding container 96424ef1ca09f8ed6dbbeaaecc1efe96887008f6a44d4c7ab69600ab237b648a: Status 404 returned error can't find the container with id 96424ef1ca09f8ed6dbbeaaecc1efe96887008f6a44d4c7ab69600ab237b648a
Nov 02 10:00:01.266009 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:01.265983    4449 generic.go:356] "Generic (PLEG): container finished" podID="b08a49e9-6d49-4ddc-9288-f6a38b620411" containerID="cd50c8e658313dee1f472e885ccb3b4358f82bfd1534b9229be1eb2e2c9bd9d0" exitCode=0
Nov 02 10:00:01.266150 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:01.266063    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v" event={"ID":"b08a49e9-6d49-4ddc-9288-f6a38b620411","Type":"ContainerDied","Data":"cd50c8e658313dee1f472e885ccb3b4358f82bfd1534b9229be1eb2e2c9bd9d0"}
Nov 02 10:00:01.266150 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:01.266123    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v" event={"ID":"b08a49e9-6d49-4ddc-9288-f6a38b620411","Type":"ContainerStarted","Data":"29e7425ed483ab4b559edb3c15714b705fbd84667ab171ff8f45de56ec192e42"}
Nov 02 10:00:01.267670 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:01.267648    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm" event={"ID":"aa87dab3-ee63-4a13-802f-af3eab2ca8f3","Type":"ContainerStarted","Data":"b1a4c7a82eb48726aed532f896830bc0fe63bf81af5b7248786f547f2cb9b354"}
Nov 02 10:00:01.267775 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:01.267684    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm" event={"ID":"aa87dab3-ee63-4a13-802f-af3eab2ca8f3","Type":"ContainerStarted","Data":"f5f3f180c4448e916d5a8504f325e0d245632effd84f70a9fb0c9edcb1b140b5"}
Nov 02 10:00:01.269258 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:01.269203    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr" event={"ID":"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec","Type":"ContainerStarted","Data":"96424ef1ca09f8ed6dbbeaaecc1efe96887008f6a44d4c7ab69600ab237b648a"}
Nov 02 10:00:01.311931 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:01.311878    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm" podStartSLOduration=1.31186363 podStartE2EDuration="1.31186363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:00:01.30877382 +0000 UTC m=+1478.247060615" watchObservedRunningTime="2026-11-02 10:00:01.31186363 +0000 UTC m=+1478.250150405"
Nov 02 10:00:01.332541 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:01.332462    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr" podStartSLOduration=1.3324467709999999 podStartE2EDuration="1.332446771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:00:01.330823873 +0000 UTC m=+1478.269110649" watchObservedRunningTime="2026-11-02 10:00:01.332446771 +0000 UTC m=+1478.270733524"
Nov 02 10:00:02.277399 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.277358    4449 generic.go:356] "Generic (PLEG): container finished" podID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3" containerID="b1a4c7a82eb48726aed532f896830bc0fe63bf81af5b7248786f547f2cb9b354" exitCode=60
Nov 02 10:00:02.277911 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.277416    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm" event={"ID":"aa87dab3-ee63-4a13-802f-af3eab2ca8f3","Type":"ContainerDied","Data":"b1a4c7a82eb48726aed532f896830bc0fe63bf81af5b7248786f547f2cb9b354"}
Nov 02 10:00:02.277911 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.277828    4449 scope.go:122] "RemoveContainer" containerID="b1a4c7a82eb48726aed532f896830bc0fe63bf81af5b7248786f547f2cb9b354"
Nov 02 10:00:02.279172 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.279148    4449 generic.go:356] "Generic (PLEG): container finished" podID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" containerID="489d9b8c64c71c1aa1438cd704b2ce192941bffef9d4e7717a926d51d6c5c86f" exitCode=60
Nov 02 10:00:02.279328 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.279280    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr" event={"ID":"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec","Type":"ContainerDied","Data":"489d9b8c64c71c1aa1438cd704b2ce192941bffef9d4e7717a926d51d6c5c86f"}
Nov 02 10:00:02.279572 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.279549    4449 scope.go:122] "RemoveContainer" containerID="489d9b8c64c71c1aa1438cd704b2ce192941bffef9d4e7717a926d51d6c5c86f"
Nov 02 10:00:02.431006 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.430981    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"
Nov 02 10:00:02.573260 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.573232    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-openshift-ca\") pod \"b08a49e9-6d49-4ddc-9288-f6a38b620411\" (UID: \"b08a49e9-6d49-4ddc-9288-f6a38b620411\") "
Nov 02 10:00:02.573438 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.573307    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-xks-ca\") pod \"b08a49e9-6d49-4ddc-9288-f6a38b620411\" (UID: \"b08a49e9-6d49-4ddc-9288-f6a38b620411\") "
Nov 02 10:00:02.573515 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.573494    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08a49e9-6d49-4ddc-9288-f6a38b620411-kube-api-access-cf5xm\" (UniqueName: \"kubernetes.io/projected/b08a49e9-6d49-4ddc-9288-f6a38b620411-kube-api-access-cf5xm\") pod \"b08a49e9-6d49-4ddc-9288-f6a38b620411\" (UID: \"b08a49e9-6d49-4ddc-9288-f6a38b620411\") "
Nov 02 10:00:02.573592 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.573565    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-openshift-ca" pod "b08a49e9-6d49-4ddc-9288-f6a38b620411" (UID: "b08a49e9-6d49-4ddc-9288-f6a38b620411"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:00:02.573859 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.573841    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:02.575493 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.575466    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-xks-ca" pod "b08a49e9-6d49-4ddc-9288-f6a38b620411" (UID: "b08a49e9-6d49-4ddc-9288-f6a38b620411"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:00:02.575642 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.575627    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08a49e9-6d49-4ddc-9288-f6a38b620411-kube-api-access-cf5xm" pod "b08a49e9-6d49-4ddc-9288-f6a38b620411" (UID: "b08a49e9-6d49-4ddc-9288-f6a38b620411"). InnerVolumeSpecName "kube-api-access-cf5xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:02.676318 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.676269    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf5xm\" (UniqueName: \"kubernetes.io/projected/b08a49e9-6d49-4ddc-9288-f6a38b620411-kube-api-access-cf5xm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:02.676318 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:02.676307    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08a49e9-6d49-4ddc-9288-f6a38b620411-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:03.287782 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:03.287749    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"
Nov 02 10:00:03.288288 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:03.287753    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v" event={"ID":"b08a49e9-6d49-4ddc-9288-f6a38b620411","Type":"ContainerDied","Data":"29e7425ed483ab4b559edb3c15714b705fbd84667ab171ff8f45de56ec192e42"}
Nov 02 10:00:03.288288 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:03.287897    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29e7425ed483ab4b559edb3c15714b705fbd84667ab171ff8f45de56ec192e42"
Nov 02 10:00:03.290587 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:03.290559    4449 generic.go:356] "Generic (PLEG): container finished" podID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3" containerID="4257a41c49384e2e04e80617c7f39cf92d8a66c732fa5229ad17298fc3c4bcd8" exitCode=60
Nov 02 10:00:03.290722 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:03.290649    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm" event={"ID":"aa87dab3-ee63-4a13-802f-af3eab2ca8f3","Type":"ContainerDied","Data":"4257a41c49384e2e04e80617c7f39cf92d8a66c732fa5229ad17298fc3c4bcd8"}
Nov 02 10:00:03.290722 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:03.290709    4449 scope.go:122] "RemoveContainer" containerID="b1a4c7a82eb48726aed532f896830bc0fe63bf81af5b7248786f547f2cb9b354"
Nov 02 10:00:03.291000 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:03.290983    4449 scope.go:122] "RemoveContainer" containerID="4257a41c49384e2e04e80617c7f39cf92d8a66c732fa5229ad17298fc3c4bcd8"
Nov 02 10:00:03.291433 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:00:03.291405    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm_odh-ai-gateway-infra(aa87dab3-ee63-4a13-802f-af3eab2ca8f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm" podUID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3"
Nov 02 10:00:03.304140 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:03.303909    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr" event={"ID":"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec","Type":"ContainerDied","Data":"195aa60981f3fd42454896f3bd9c1e97e2425e1b6185d5e9ba22adb34921e18a"}
Nov 02 10:00:03.304140 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:03.304086    4449 scope.go:122] "RemoveContainer" containerID="195aa60981f3fd42454896f3bd9c1e97e2425e1b6185d5e9ba22adb34921e18a"
Nov 02 10:00:03.307230 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:03.304144    4449 generic.go:356] "Generic (PLEG): container finished" podID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" containerID="195aa60981f3fd42454896f3bd9c1e97e2425e1b6185d5e9ba22adb34921e18a" exitCode=60
Nov 02 10:00:03.307230 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:00:03.304686    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893560-tr7jr_odh-ai-gateway-infra(da0e46e3-ae58-4909-8eaa-e57bbb0a62ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr" podUID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec"
Nov 02 10:00:03.326039 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:03.326006    4449 scope.go:122] "RemoveContainer" containerID="489d9b8c64c71c1aa1438cd704b2ce192941bffef9d4e7717a926d51d6c5c86f"
Nov 02 10:00:04.313665 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:04.313633    4449 scope.go:122] "RemoveContainer" containerID="4257a41c49384e2e04e80617c7f39cf92d8a66c732fa5229ad17298fc3c4bcd8"
Nov 02 10:00:04.314115 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:00:04.313926    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm_odh-ai-gateway-infra(aa87dab3-ee63-4a13-802f-af3eab2ca8f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm" podUID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3"
Nov 02 10:00:04.315201 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:04.315183    4449 scope.go:122] "RemoveContainer" containerID="195aa60981f3fd42454896f3bd9c1e97e2425e1b6185d5e9ba22adb34921e18a"
Nov 02 10:00:04.315420 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:00:04.315402    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893560-tr7jr_odh-ai-gateway-infra(da0e46e3-ae58-4909-8eaa-e57bbb0a62ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr" podUID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec"
Nov 02 10:00:06.547685 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:00:06.547647    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:00:14.200100 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:14.200057    4449 scope.go:122] "RemoveContainer" containerID="4257a41c49384e2e04e80617c7f39cf92d8a66c732fa5229ad17298fc3c4bcd8"
Nov 02 10:00:15.437600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:15.437559    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm" event={"ID":"aa87dab3-ee63-4a13-802f-af3eab2ca8f3","Type":"ContainerStarted","Data":"3d8320a3c6a601de0fd377c75c536e7d79c14969ccc9851640cd665ebe52e4dd"}
Nov 02 10:00:16.447907 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:16.447819    4449 generic.go:356] "Generic (PLEG): container finished" podID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3" containerID="3d8320a3c6a601de0fd377c75c536e7d79c14969ccc9851640cd665ebe52e4dd" exitCode=60
Nov 02 10:00:16.447907 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:16.447883    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm" event={"ID":"aa87dab3-ee63-4a13-802f-af3eab2ca8f3","Type":"ContainerDied","Data":"3d8320a3c6a601de0fd377c75c536e7d79c14969ccc9851640cd665ebe52e4dd"}
Nov 02 10:00:16.448416 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:16.447930    4449 scope.go:122] "RemoveContainer" containerID="4257a41c49384e2e04e80617c7f39cf92d8a66c732fa5229ad17298fc3c4bcd8"
Nov 02 10:00:16.448416 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:16.448321    4449 scope.go:122] "RemoveContainer" containerID="3d8320a3c6a601de0fd377c75c536e7d79c14969ccc9851640cd665ebe52e4dd"
Nov 02 10:00:16.448676 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:00:16.448648    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm_odh-ai-gateway-infra(aa87dab3-ee63-4a13-802f-af3eab2ca8f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm" podUID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3"
Nov 02 10:00:16.478134 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:16.478101    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"]
Nov 02 10:00:17.200143 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:17.200107    4449 scope.go:122] "RemoveContainer" containerID="195aa60981f3fd42454896f3bd9c1e97e2425e1b6185d5e9ba22adb34921e18a"
Nov 02 10:00:17.623631 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:17.623602    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"
Nov 02 10:00:17.767940 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:17.767840    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-kube-api-access-wmt58\" (UniqueName: \"kubernetes.io/projected/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-kube-api-access-wmt58\") pod \"aa87dab3-ee63-4a13-802f-af3eab2ca8f3\" (UID: \"aa87dab3-ee63-4a13-802f-af3eab2ca8f3\") "
Nov 02 10:00:17.767940 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:17.767898    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-maas-api-service-ca\") pod \"aa87dab3-ee63-4a13-802f-af3eab2ca8f3\" (UID: \"aa87dab3-ee63-4a13-802f-af3eab2ca8f3\") "
Nov 02 10:00:17.768302 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:17.768275    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-maas-api-service-ca" pod "aa87dab3-ee63-4a13-802f-af3eab2ca8f3" (UID: "aa87dab3-ee63-4a13-802f-af3eab2ca8f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:00:17.770206 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:17.770178    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-kube-api-access-wmt58" pod "aa87dab3-ee63-4a13-802f-af3eab2ca8f3" (UID: "aa87dab3-ee63-4a13-802f-af3eab2ca8f3"). InnerVolumeSpecName "kube-api-access-wmt58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:17.870001 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:17.869958    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmt58\" (UniqueName: \"kubernetes.io/projected/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-kube-api-access-wmt58\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:17.870001 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:17.870003    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa87dab3-ee63-4a13-802f-af3eab2ca8f3-maas-api-service-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:18.466928 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:18.466902    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"
Nov 02 10:00:18.466928 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:18.466907    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm" event={"ID":"aa87dab3-ee63-4a13-802f-af3eab2ca8f3","Type":"ContainerDied","Data":"f5f3f180c4448e916d5a8504f325e0d245632effd84f70a9fb0c9edcb1b140b5"}
Nov 02 10:00:18.467177 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:18.466967    4449 scope.go:122] "RemoveContainer" containerID="3d8320a3c6a601de0fd377c75c536e7d79c14969ccc9851640cd665ebe52e4dd"
Nov 02 10:00:18.469210 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:18.469181    4449 generic.go:356] "Generic (PLEG): container finished" podID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" containerID="a38423dcd0084cea2e43e99f1b3d16d679dda8321234ea27db0cda05a56960f5" exitCode=60
Nov 02 10:00:18.469362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:18.469342    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr" event={"ID":"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec","Type":"ContainerDied","Data":"a38423dcd0084cea2e43e99f1b3d16d679dda8321234ea27db0cda05a56960f5"}
Nov 02 10:00:18.469774 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:18.469759    4449 scope.go:122] "RemoveContainer" containerID="a38423dcd0084cea2e43e99f1b3d16d679dda8321234ea27db0cda05a56960f5"
Nov 02 10:00:18.470094 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:00:18.470075    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893560-tr7jr_odh-ai-gateway-infra(da0e46e3-ae58-4909-8eaa-e57bbb0a62ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr" podUID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec"
Nov 02 10:00:18.479194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:18.479177    4449 scope.go:122] "RemoveContainer" containerID="195aa60981f3fd42454896f3bd9c1e97e2425e1b6185d5e9ba22adb34921e18a"
Nov 02 10:00:18.507678 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:18.507646    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"]
Nov 02 10:00:18.511648 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:18.511580    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-629129-29893560-662qm"]
Nov 02 10:00:19.217376 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:19.217338    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3" path="/var/lib/kubelet/pods/aa87dab3-ee63-4a13-802f-af3eab2ca8f3/volumes"
Nov 02 10:00:19.501983 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:19.501899    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"]
Nov 02 10:00:19.650207 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:19.650180    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"
Nov 02 10:00:19.795196 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:19.795164    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-kube-api-access-sxjrv\" (UniqueName: \"kubernetes.io/projected/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-kube-api-access-sxjrv\") pod \"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec\" (UID: \"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec\") "
Nov 02 10:00:19.795417 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:19.795282    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-maas-api-service-ca\") pod \"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec\" (UID: \"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec\") "
Nov 02 10:00:19.795722 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:19.795691    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-maas-api-service-ca" pod "da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" (UID: "da0e46e3-ae58-4909-8eaa-e57bbb0a62ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:00:19.811742 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:19.811686    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-kube-api-access-sxjrv" pod "da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" (UID: "da0e46e3-ae58-4909-8eaa-e57bbb0a62ec"). InnerVolumeSpecName "kube-api-access-sxjrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:19.897251 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:19.897177    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxjrv\" (UniqueName: \"kubernetes.io/projected/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-kube-api-access-sxjrv\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:19.897251 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:19.897208    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec-maas-api-service-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:20.488468 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:20.488432    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"
Nov 02 10:00:20.488963 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:20.488467    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr" event={"ID":"da0e46e3-ae58-4909-8eaa-e57bbb0a62ec","Type":"ContainerDied","Data":"96424ef1ca09f8ed6dbbeaaecc1efe96887008f6a44d4c7ab69600ab237b648a"}
Nov 02 10:00:20.488963 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:20.488534    4449 scope.go:122] "RemoveContainer" containerID="a38423dcd0084cea2e43e99f1b3d16d679dda8321234ea27db0cda05a56960f5"
Nov 02 10:00:20.521514 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:20.521477    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"]
Nov 02 10:00:20.529066 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:20.529035    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893560-tr7jr"]
Nov 02 10:00:21.206719 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:21.206672    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" path="/var/lib/kubelet/pods/da0e46e3-ae58-4909-8eaa-e57bbb0a62ec/volumes"
Nov 02 10:00:25.431969 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:25.431930    4449 scope.go:122] "RemoveContainer" containerID="93f1dcf8ca6df1ce4171274c8b8244a6abce6448fa6c0cfb258341ea27ea4638"
Nov 02 10:00:25.442692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:25.442667    4449 scope.go:122] "RemoveContainer" containerID="177c7ebd69ab47d557c341f66a71146b3d8a40cc30998049800dfe51d6f64857"
Nov 02 10:00:25.453490 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:25.453464    4449 scope.go:122] "RemoveContainer" containerID="39e5aa0b819f2923c1dbf1910e39a1446dcaf1bd109cf2b24957c8d98052990c"
Nov 02 10:00:25.464273 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:25.464250    4449 scope.go:122] "RemoveContainer" containerID="8906a431f2c657b81583c2e745b2d529393b760b9a857dc5e137aab254ac4f84"
Nov 02 10:00:25.475733 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:25.475708    4449 scope.go:122] "RemoveContainer" containerID="66e93384ea42b4c71d908e7983d8ab4fb5bac2ba93052bd115d57b755310a453"
Nov 02 10:00:25.486090 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:25.486065    4449 scope.go:122] "RemoveContainer" containerID="7492b6523876f2e7aa2fd1d0c6d80a0646c1dddab65e1e98fadb6d1bfd82d53e"
Nov 02 10:00:25.496393 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:25.496370    4449 scope.go:122] "RemoveContainer" containerID="c4b7dd3d68c0552c8fdff88314ac6e5bd4b3db9a0634c721cd71d5b2aea1ffd4"
Nov 02 10:00:25.510126 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:25.510084    4449 scope.go:122] "RemoveContainer" containerID="772a22865c7b0d95f94899ec8031b351cfc72121367b1039160afcf991d94536"
Nov 02 10:00:25.540314 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:25.537356    4449 scope.go:122] "RemoveContainer" containerID="9a8406f2ad95b729c5650ac0dc15d58456069dd213d5ab5a6b4e06dcc9fb34d8"
Nov 02 10:00:29.026898 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:29.026862    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"]
Nov 02 10:00:29.030688 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:29.030659    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d76ecdd-d7rmq"]
Nov 02 10:00:29.224550 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:29.224007    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230a22f3-5692-4bf8-8897-850431c806ec" path="/var/lib/kubelet/pods/230a22f3-5692-4bf8-8897-850431c806ec/volumes"
Nov 02 10:00:36.547567 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:00:36.547529    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:00:43.033483 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:43.033440    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"]
Nov 02 10:00:43.037469 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:43.037438    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ced927a-74m87"]
Nov 02 10:00:43.207197 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:43.207160    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26636211-ba44-4e02-9407-2f65b0503f72" path="/var/lib/kubelet/pods/26636211-ba44-4e02-9407-2f65b0503f72/volumes"
Nov 02 10:00:45.028264 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:45.028190    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"]
Nov 02 10:00:45.032758 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:45.032723    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7458c4da-rjbr6"]
Nov 02 10:00:45.223893 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:45.223850    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4560961e-36b1-4312-96d2-371f0e228c73" path="/var/lib/kubelet/pods/4560961e-36b1-4312-96d2-371f0e228c73/volumes"
Nov 02 10:00:52.026971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:52.026930    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"]
Nov 02 10:00:52.030647 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:52.030622    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2af7639a-8dzf2"]
Nov 02 10:00:53.206351 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:00:53.206317    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0" path="/var/lib/kubelet/pods/3157e9f7-d8a6-4ce2-81e1-07337f6d9bd0/volumes"
Nov 02 10:01:03.024992 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.024950    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"]
Nov 02 10:01:03.028714 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.028687    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb44f15-l64j4"]
Nov 02 10:01:03.206032 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.206000    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebbb08be-db62-47c4-8d7b-6eb3a3598438" path="/var/lib/kubelet/pods/ebbb08be-db62-47c4-8d7b-6eb3a3598438/volumes"
Nov 02 10:01:03.745505 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.745468    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"]
Nov 02 10:01:03.746001 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.745988    4449 cpu_manager.go:414] "containerMap: removing container" podUID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" containerName="cleanup"
Nov 02 10:01:03.746047 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746003    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" containerName="cleanup"
Nov 02 10:01:03.746047 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746025    4449 cpu_manager.go:414] "containerMap: removing container" podUID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3" containerName="cleanup"
Nov 02 10:01:03.746047 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746030    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3" containerName="cleanup"
Nov 02 10:01:03.746047 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746044    4449 cpu_manager.go:414] "containerMap: removing container" podUID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3" containerName="cleanup"
Nov 02 10:01:03.746181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746050    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3" containerName="cleanup"
Nov 02 10:01:03.746181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746072    4449 cpu_manager.go:414] "containerMap: removing container" podUID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" containerName="cleanup"
Nov 02 10:01:03.746181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746076    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" containerName="cleanup"
Nov 02 10:01:03.746181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746084    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b08a49e9-6d49-4ddc-9288-f6a38b620411" containerName="revoke-keys"
Nov 02 10:01:03.746181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746088    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08a49e9-6d49-4ddc-9288-f6a38b620411" containerName="revoke-keys"
Nov 02 10:01:03.746181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746165    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3" containerName="cleanup"
Nov 02 10:01:03.746181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746173    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3" containerName="cleanup"
Nov 02 10:01:03.746181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746182    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" containerName="cleanup"
Nov 02 10:01:03.746470 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746187    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" containerName="cleanup"
Nov 02 10:01:03.746470 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746194    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08a49e9-6d49-4ddc-9288-f6a38b620411" containerName="revoke-keys"
Nov 02 10:01:03.746470 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.746200    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" containerName="cleanup"
Nov 02 10:01:03.750829 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.750812    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"
Nov 02 10:01:03.759860 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.759837    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 10:01:03.759970 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.759936    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:01:03.760040 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.760014    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:01:03.760115 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.760101    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:01:03.760375 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.760354    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"]
Nov 02 10:01:03.918357 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.918322    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30607e68-7tmmg\" (UID: \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"
Nov 02 10:01:03.918540 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.918443    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30607e68-7tmmg\" (UID: \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"
Nov 02 10:01:03.918595 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:03.918582    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79vnh\" (UniqueName: \"kubernetes.io/projected/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-kube-api-access-79vnh\") pod \"maas-api-revoke-sub-30607e68-7tmmg\" (UID: \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"
Nov 02 10:01:04.020194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:04.020104    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30607e68-7tmmg\" (UID: \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"
Nov 02 10:01:04.020388 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:04.020255    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79vnh\" (UniqueName: \"kubernetes.io/projected/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-kube-api-access-79vnh\") pod \"maas-api-revoke-sub-30607e68-7tmmg\" (UID: \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"
Nov 02 10:01:04.020388 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:04.020329    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30607e68-7tmmg\" (UID: \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"
Nov 02 10:01:04.020992 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:04.020970    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30607e68-7tmmg\" (UID: \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"
Nov 02 10:01:04.022825 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:04.022790    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30607e68-7tmmg\" (UID: \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"
Nov 02 10:01:04.038550 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:04.038522    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79vnh\" (UniqueName: \"kubernetes.io/projected/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-kube-api-access-79vnh\") pod \"maas-api-revoke-sub-30607e68-7tmmg\" (UID: \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"
Nov 02 10:01:04.066584 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:04.066538    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"
Nov 02 10:01:04.349903 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:04.349865    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"]
Nov 02 10:01:04.350606 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:01:04.350580    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d13c536_fd9e_4ed4_9adb_cdd0a5d1fd3d.slice/crio-e87f36065b567505acdb8dc3d82f8c88ae713f92e1eed8a9e20803f9b7f7f0a8 WatchSource:0}: Error finding container e87f36065b567505acdb8dc3d82f8c88ae713f92e1eed8a9e20803f9b7f7f0a8: Status 404 returned error can't find the container with id e87f36065b567505acdb8dc3d82f8c88ae713f92e1eed8a9e20803f9b7f7f0a8
Nov 02 10:01:04.872438 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:04.872393    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg" event={"ID":"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d","Type":"ContainerStarted","Data":"fabedce7d42576ac1964d03e069e375a8cd7c98946e863c9fe085b41e116acdb"}
Nov 02 10:01:04.872678 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:04.872502    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg" event={"ID":"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d","Type":"ContainerStarted","Data":"e87f36065b567505acdb8dc3d82f8c88ae713f92e1eed8a9e20803f9b7f7f0a8"}
Nov 02 10:01:04.894398 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:04.894338    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg" podStartSLOduration=1.894322238 podStartE2EDuration="1.894322238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:01:04.89154099 +0000 UTC m=+1541.829827794" watchObservedRunningTime="2026-11-02 10:01:04.894322238 +0000 UTC m=+1541.832609015"
Nov 02 10:01:05.881310 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:05.881274    4449 generic.go:356] "Generic (PLEG): container finished" podID="8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d" containerID="fabedce7d42576ac1964d03e069e375a8cd7c98946e863c9fe085b41e116acdb" exitCode=0
Nov 02 10:01:05.883924 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:05.881365    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg" event={"ID":"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d","Type":"ContainerDied","Data":"fabedce7d42576ac1964d03e069e375a8cd7c98946e863c9fe085b41e116acdb"}
Nov 02 10:01:06.547584 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:01:06.547540    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:01:07.054229 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:07.054194    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"
Nov 02 10:01:07.163628 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:07.163534    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-xks-ca\") pod \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\" (UID: \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\") "
Nov 02 10:01:07.163823 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:07.163630    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-openshift-ca\") pod \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\" (UID: \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\") "
Nov 02 10:01:07.163823 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:07.163687    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-kube-api-access-79vnh\" (UniqueName: \"kubernetes.io/projected/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-kube-api-access-79vnh\") pod \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\" (UID: \"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d\") "
Nov 02 10:01:07.164067 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:07.164039    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-openshift-ca" pod "8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d" (UID: "8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:01:07.165832 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:07.165805    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-xks-ca" pod "8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d" (UID: "8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:01:07.165986 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:07.165961    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-kube-api-access-79vnh" pod "8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d" (UID: "8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d"). InnerVolumeSpecName "kube-api-access-79vnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:07.265177 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:07.265141    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:07.265177 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:07.265174    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:07.265177 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:07.265185    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79vnh\" (UniqueName: \"kubernetes.io/projected/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d-kube-api-access-79vnh\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:07.895448 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:07.895420    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"
Nov 02 10:01:07.895724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:07.895446    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg" event={"ID":"8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d","Type":"ContainerDied","Data":"e87f36065b567505acdb8dc3d82f8c88ae713f92e1eed8a9e20803f9b7f7f0a8"}
Nov 02 10:01:07.895724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:07.895493    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e87f36065b567505acdb8dc3d82f8c88ae713f92e1eed8a9e20803f9b7f7f0a8"
Nov 02 10:01:08.280589 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:08.280494    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"]
Nov 02 10:01:08.744842 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:08.744800    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"]
Nov 02 10:01:08.745053 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:08.745009    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerName="istio-proxy" containerID="cri-o://76921ab24f098a1f4d6601045e75474b37ddc61699dddc97b7a409fcd2825720" gracePeriod=30
Nov 02 10:01:13.950500 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:13.950452    4449 generic.go:356] "Generic (PLEG): container finished" podID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerID="76921ab24f098a1f4d6601045e75474b37ddc61699dddc97b7a409fcd2825720" exitCode=0
Nov 02 10:01:13.951020 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:13.950508    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" event={"ID":"0c7b8d80-80a7-47b3-876d-dce3ebb293fa","Type":"ContainerDied","Data":"76921ab24f098a1f4d6601045e75474b37ddc61699dddc97b7a409fcd2825720"}
Nov 02 10:01:14.028363 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.028290    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 10:01:14.048886 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.048833    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-envoy\") pod \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") "
Nov 02 10:01:14.049130 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.048915    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-credential-socket\") pod \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") "
Nov 02 10:01:14.049130 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.048940    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-socket\") pod \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") "
Nov 02 10:01:14.049130 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.048982    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-data\") pod \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") "
Nov 02 10:01:14.049130 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.049013    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-token\" (UniqueName: \"kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-token\") pod \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") "
Nov 02 10:01:14.049130 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.049103    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istiod-ca-cert\") pod \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") "
Nov 02 10:01:14.049464 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.049265    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-credential-socket" pod "0c7b8d80-80a7-47b3-876d-dce3ebb293fa" (UID: "0c7b8d80-80a7-47b3-876d-dce3ebb293fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:14.049464 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.049341    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-data" pod "0c7b8d80-80a7-47b3-876d-dce3ebb293fa" (UID: "0c7b8d80-80a7-47b3-876d-dce3ebb293fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:14.049464 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.049357    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-socket" pod "0c7b8d80-80a7-47b3-876d-dce3ebb293fa" (UID: "0c7b8d80-80a7-47b3-876d-dce3ebb293fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:14.049628 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.049507    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istiod-ca-cert" pod "0c7b8d80-80a7-47b3-876d-dce3ebb293fa" (UID: "0c7b8d80-80a7-47b3-876d-dce3ebb293fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:01:14.049628 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.049538    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-podinfo\") pod \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") "
Nov 02 10:01:14.049628 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.049613    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-kube-api-access-nqr4k\" (UniqueName: \"kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-kube-api-access-nqr4k\") pod \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") "
Nov 02 10:01:14.049808 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.049661    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-certs\") pod \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\" (UID: \"0c7b8d80-80a7-47b3-876d-dce3ebb293fa\") "
Nov 02 10:01:14.050052 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.049979    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-certs" pod "0c7b8d80-80a7-47b3-876d-dce3ebb293fa" (UID: "0c7b8d80-80a7-47b3-876d-dce3ebb293fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:14.050560 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.050535    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:14.050675 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.050565    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:14.050675 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.050581    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:14.050675 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.050598    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:14.050675 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.050612    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:14.052477 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.052425    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-podinfo" pod "0c7b8d80-80a7-47b3-876d-dce3ebb293fa" (UID: "0c7b8d80-80a7-47b3-876d-dce3ebb293fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:01:14.052588 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.052544    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-envoy" pod "0c7b8d80-80a7-47b3-876d-dce3ebb293fa" (UID: "0c7b8d80-80a7-47b3-876d-dce3ebb293fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:14.052960 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.052938    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-kube-api-access-nqr4k" pod "0c7b8d80-80a7-47b3-876d-dce3ebb293fa" (UID: "0c7b8d80-80a7-47b3-876d-dce3ebb293fa"). InnerVolumeSpecName "kube-api-access-nqr4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:14.053073 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.052980    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-token" pod "0c7b8d80-80a7-47b3-876d-dce3ebb293fa" (UID: "0c7b8d80-80a7-47b3-876d-dce3ebb293fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:14.151930 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.151882    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:14.151930 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.151923    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:14.151930 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.151935    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqr4k\" (UniqueName: \"kubernetes.io/projected/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-kube-api-access-nqr4k\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:14.152176 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.151946    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7b8d80-80a7-47b3-876d-dce3ebb293fa-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:14.958997 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.958961    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j" event={"ID":"0c7b8d80-80a7-47b3-876d-dce3ebb293fa","Type":"ContainerDied","Data":"13f67a8976035656d4116ca078e5e40a8ae3c5b6de1e0eaa9f3a80ec9e6e30ff"}
Nov 02 10:01:14.959465 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.959012    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"
Nov 02 10:01:14.959465 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.959057    4449 scope.go:122] "RemoveContainer" containerID="76921ab24f098a1f4d6601045e75474b37ddc61699dddc97b7a409fcd2825720"
Nov 02 10:01:14.987321 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.987000    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"]
Nov 02 10:01:14.990934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:14.990901    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-dg56j"]
Nov 02 10:01:15.206741 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.206707    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" path="/var/lib/kubelet/pods/0c7b8d80-80a7-47b3-876d-dce3ebb293fa/volumes"
Nov 02 10:01:15.358920 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.357863    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"]
Nov 02 10:01:15.663632 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.663547    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"]
Nov 02 10:01:15.664657 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.664629    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerName="istio-proxy"
Nov 02 10:01:15.664882 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.664865    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerName="istio-proxy"
Nov 02 10:01:15.665016 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.665004    4449 cpu_manager.go:414] "containerMap: removing container" podUID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" containerName="cleanup"
Nov 02 10:01:15.665128 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.665115    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0e46e3-ae58-4909-8eaa-e57bbb0a62ec" containerName="cleanup"
Nov 02 10:01:15.665334 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.665318    4449 cpu_manager.go:414] "containerMap: removing container" podUID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3" containerName="cleanup"
Nov 02 10:01:15.665451 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.665439    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3" containerName="cleanup"
Nov 02 10:01:15.665540 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.665530    4449 cpu_manager.go:414] "containerMap: removing container" podUID="8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d" containerName="revoke-keys"
Nov 02 10:01:15.665612 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.665602    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d" containerName="revoke-keys"
Nov 02 10:01:15.665945 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.665845    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d" containerName="revoke-keys"
Nov 02 10:01:15.666086 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.666070    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa87dab3-ee63-4a13-802f-af3eab2ca8f3" containerName="cleanup"
Nov 02 10:01:15.666183 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.666173    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c7b8d80-80a7-47b3-876d-dce3ebb293fa" containerName="istio-proxy"
Nov 02 10:01:15.677166 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.677138    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"
Nov 02 10:01:15.684247 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.683634    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:01:15.684247 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.683660    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:01:15.684247 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.683739    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:01:15.684247 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.683824    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 10:01:15.701288 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.701258    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"]
Nov 02 10:01:15.771993 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.771948    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhzbf\" (UniqueName: \"kubernetes.io/projected/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-kube-api-access-qhzbf\") pod \"maas-api-revoke-sub-5ad9d226-m9qbt\" (UID: \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"
Nov 02 10:01:15.772183 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.772009    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ad9d226-m9qbt\" (UID: \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"
Nov 02 10:01:15.772183 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.772164    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ad9d226-m9qbt\" (UID: \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"
Nov 02 10:01:15.873078 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.873041    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ad9d226-m9qbt\" (UID: \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"
Nov 02 10:01:15.873360 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.873199    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhzbf\" (UniqueName: \"kubernetes.io/projected/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-kube-api-access-qhzbf\") pod \"maas-api-revoke-sub-5ad9d226-m9qbt\" (UID: \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"
Nov 02 10:01:15.873360 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.873316    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ad9d226-m9qbt\" (UID: \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"
Nov 02 10:01:15.873940 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.873921    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ad9d226-m9qbt\" (UID: \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"
Nov 02 10:01:15.875645 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.875614    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ad9d226-m9qbt\" (UID: \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"
Nov 02 10:01:15.886379 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:15.886350    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhzbf\" (UniqueName: \"kubernetes.io/projected/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-kube-api-access-qhzbf\") pod \"maas-api-revoke-sub-5ad9d226-m9qbt\" (UID: \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"
Nov 02 10:01:16.003413 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:16.003322    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"
Nov 02 10:01:16.033244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:16.033185    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"]
Nov 02 10:01:16.034948 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:16.034925    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4451c93-9tfzm"]
Nov 02 10:01:16.295492 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:16.295461    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"]
Nov 02 10:01:16.296494 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:01:16.296459    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff3cbe7_38dd_4f32_a0ad_8ca2ac7a8c57.slice/crio-b5457a3d41c816dfb26e2874f029a1f13b30ba61101f1746d98ea44d2c454fcb WatchSource:0}: Error finding container b5457a3d41c816dfb26e2874f029a1f13b30ba61101f1746d98ea44d2c454fcb: Status 404 returned error can't find the container with id b5457a3d41c816dfb26e2874f029a1f13b30ba61101f1746d98ea44d2c454fcb
Nov 02 10:01:16.977779 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:16.977692    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt" event={"ID":"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57","Type":"ContainerStarted","Data":"f6146e0b8200e52c0cb41f9949e7b51d5fedbf9f72109b0c66a59d7671ea6460"}
Nov 02 10:01:16.977779 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:16.977746    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt" event={"ID":"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57","Type":"ContainerStarted","Data":"b5457a3d41c816dfb26e2874f029a1f13b30ba61101f1746d98ea44d2c454fcb"}
Nov 02 10:01:17.003179 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:17.003124    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt" podStartSLOduration=2.003108837 podStartE2EDuration="2.003108837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:01:17.000189306 +0000 UTC m=+1553.938476082" watchObservedRunningTime="2026-11-02 10:01:17.003108837 +0000 UTC m=+1553.941395613"
Nov 02 10:01:17.220642 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:17.220602    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7169063e-63b8-4f1b-bfba-2d612ab7627b" path="/var/lib/kubelet/pods/7169063e-63b8-4f1b-bfba-2d612ab7627b/volumes"
Nov 02 10:01:17.987655 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:17.987611    4449 generic.go:356] "Generic (PLEG): container finished" podID="9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57" containerID="f6146e0b8200e52c0cb41f9949e7b51d5fedbf9f72109b0c66a59d7671ea6460" exitCode=0
Nov 02 10:01:17.987871 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:17.987656    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt" event={"ID":"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57","Type":"ContainerDied","Data":"f6146e0b8200e52c0cb41f9949e7b51d5fedbf9f72109b0c66a59d7671ea6460"}
Nov 02 10:01:19.143390 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:19.143359    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"
Nov 02 10:01:19.230858 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:19.230821    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-xks-ca\") pod \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\" (UID: \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\") "
Nov 02 10:01:19.231051 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:19.230886    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-kube-api-access-qhzbf\" (UniqueName: \"kubernetes.io/projected/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-kube-api-access-qhzbf\") pod \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\" (UID: \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\") "
Nov 02 10:01:19.231103 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:19.231066    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-openshift-ca\") pod \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\" (UID: \"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57\") "
Nov 02 10:01:19.231522 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:19.231488    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-openshift-ca" pod "9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57" (UID: "9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:01:19.233276 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:19.233254    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-kube-api-access-qhzbf" pod "9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57" (UID: "9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57"). InnerVolumeSpecName "kube-api-access-qhzbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:19.233343 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:19.233253    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-xks-ca" pod "9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57" (UID: "9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:01:19.333184 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:19.333148    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhzbf\" (UniqueName: \"kubernetes.io/projected/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-kube-api-access-qhzbf\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:19.333184 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:19.333185    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:19.333422 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:19.333201    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:20.004267 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:20.004210    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"
Nov 02 10:01:20.004267 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:20.004245    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt" event={"ID":"9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57","Type":"ContainerDied","Data":"b5457a3d41c816dfb26e2874f029a1f13b30ba61101f1746d98ea44d2c454fcb"}
Nov 02 10:01:20.004506 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:20.004298    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5457a3d41c816dfb26e2874f029a1f13b30ba61101f1746d98ea44d2c454fcb"
Nov 02 10:01:20.358425 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:20.358374    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh" podUID="6d32044d-b33f-4d9d-8e4a-c974bb485074" containerName="payload-pre-processing" containerID="cri-o://5df174620b4cd75f10c19d06a5db9d97cd3a5bcde6098dd82d111552c5f6ea56" gracePeriod=25
Nov 02 10:01:23.301981 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.301940    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" containerID="cri-o://ac46986bdef3856072a0ab53c7e690a75d9755efb227605139d3ea1dec80bbcc" gracePeriod=45
Nov 02 10:01:23.566310 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.566204    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 10:01:23.711653 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.711614    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-home\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-home\") pod \"cb3033ba-0513-428c-9380-50d8f9f3bca8\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") "
Nov 02 10:01:23.711924 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.711698    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-tmp-dir\") pod \"cb3033ba-0513-428c-9380-50d8f9f3bca8\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") "
Nov 02 10:01:23.711924 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.711783    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-dshm\") pod \"cb3033ba-0513-428c-9380-50d8f9f3bca8\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") "
Nov 02 10:01:23.711924 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.711899    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-home" pod "cb3033ba-0513-428c-9380-50d8f9f3bca8" (UID: "cb3033ba-0513-428c-9380-50d8f9f3bca8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.712131 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.711982    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb3033ba-0513-428c-9380-50d8f9f3bca8-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3033ba-0513-428c-9380-50d8f9f3bca8-tls-certs\") pod \"cb3033ba-0513-428c-9380-50d8f9f3bca8\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") "
Nov 02 10:01:23.712131 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.712014    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-model-cache\") pod \"cb3033ba-0513-428c-9380-50d8f9f3bca8\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") "
Nov 02 10:01:23.712131 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.712046    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3033ba-0513-428c-9380-50d8f9f3bca8-kube-api-access-8sjcq\" (UniqueName: \"kubernetes.io/projected/cb3033ba-0513-428c-9380-50d8f9f3bca8-kube-api-access-8sjcq\") pod \"cb3033ba-0513-428c-9380-50d8f9f3bca8\" (UID: \"cb3033ba-0513-428c-9380-50d8f9f3bca8\") "
Nov 02 10:01:23.712548 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.712489    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-tmp-dir" pod "cb3033ba-0513-428c-9380-50d8f9f3bca8" (UID: "cb3033ba-0513-428c-9380-50d8f9f3bca8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.712791 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.712597    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-model-cache" pod "cb3033ba-0513-428c-9380-50d8f9f3bca8" (UID: "cb3033ba-0513-428c-9380-50d8f9f3bca8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.713115 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.713083    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.713274 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.713141    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.713274 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.713158    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.714611 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.714573    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-dshm" pod "cb3033ba-0513-428c-9380-50d8f9f3bca8" (UID: "cb3033ba-0513-428c-9380-50d8f9f3bca8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.715197 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.715166    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3033ba-0513-428c-9380-50d8f9f3bca8-kube-api-access-8sjcq" pod "cb3033ba-0513-428c-9380-50d8f9f3bca8" (UID: "cb3033ba-0513-428c-9380-50d8f9f3bca8"). InnerVolumeSpecName "kube-api-access-8sjcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:23.715446 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.715423    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3033ba-0513-428c-9380-50d8f9f3bca8-tls-certs" pod "cb3033ba-0513-428c-9380-50d8f9f3bca8" (UID: "cb3033ba-0513-428c-9380-50d8f9f3bca8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:01:23.811520 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.811476    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"]
Nov 02 10:01:23.812038 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.812023    4449 cpu_manager.go:414] "containerMap: removing container" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main"
Nov 02 10:01:23.812038 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.812038    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main"
Nov 02 10:01:23.812169 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.812068    4449 cpu_manager.go:414] "containerMap: removing container" podUID="9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57" containerName="revoke-keys"
Nov 02 10:01:23.812169 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.812073    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57" containerName="revoke-keys"
Nov 02 10:01:23.812169 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.812154    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57" containerName="revoke-keys"
Nov 02 10:01:23.812169 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.812166    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main"
Nov 02 10:01:23.814943 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.814910    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3033ba-0513-428c-9380-50d8f9f3bca8-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.815108 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.814946    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sjcq\" (UniqueName: \"kubernetes.io/projected/cb3033ba-0513-428c-9380-50d8f9f3bca8-kube-api-access-8sjcq\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.815108 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.814966    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb3033ba-0513-428c-9380-50d8f9f3bca8-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.816045 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.816021    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:23.835547 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.835510    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"]
Nov 02 10:01:23.916865 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.916797    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:23.917336 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.916869    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:23.917336 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.916970    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:23.917336 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.917075    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:23.917336 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.917130    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:23.917336 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.917288    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zknpv\" (UniqueName: \"kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-kube-api-access-zknpv\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:23.917822 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.917485    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:23.917822 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.917514    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:23.917822 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:23.917539    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.018940 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.018893    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.018940 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.018942    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.019186 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.018962    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.019186 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.019096    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.019186 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.019150    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.019361 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.019322    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.019424 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.019367    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.019424 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.019413    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.019542 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.019450    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknpv\" (UniqueName: \"kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-kube-api-access-zknpv\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.019542 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.019496    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.019771 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.019739    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.019953 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.019935    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.020043 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.020018    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.020685 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.020660    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.021556 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.021536    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.021888 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.021856    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.032781 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.032741    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"]
Nov 02 10:01:24.034746 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.034717    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zknpv\" (UniqueName: \"kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-kube-api-access-zknpv\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.036467 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.036436    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.036655 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.036636    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac98a2-nfjx8"]
Nov 02 10:01:24.037434 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.037411    4449 generic.go:356] "Generic (PLEG): container finished" podID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerID="ac46986bdef3856072a0ab53c7e690a75d9755efb227605139d3ea1dec80bbcc" exitCode=0
Nov 02 10:01:24.037537 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.037452    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"
Nov 02 10:01:24.037537 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.037476    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" event={"ID":"cb3033ba-0513-428c-9380-50d8f9f3bca8","Type":"ContainerDied","Data":"ac46986bdef3856072a0ab53c7e690a75d9755efb227605139d3ea1dec80bbcc"}
Nov 02 10:01:24.037537 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.037508    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" event={"ID":"cb3033ba-0513-428c-9380-50d8f9f3bca8","Type":"ContainerDied","Data":"39b7c914d8972c5e0d637fd9f76dba4612ddf6658b9048a38d02439a6551c61b"}
Nov 02 10:01:24.037659 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.037525    4449 scope.go:122] "RemoveContainer" containerID="ac46986bdef3856072a0ab53c7e690a75d9755efb227605139d3ea1dec80bbcc"
Nov 02 10:01:24.052427 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.052402    4449 scope.go:122] "RemoveContainer" containerID="ac46986bdef3856072a0ab53c7e690a75d9755efb227605139d3ea1dec80bbcc"
Nov 02 10:01:24.052975 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:01:24.052949    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac46986bdef3856072a0ab53c7e690a75d9755efb227605139d3ea1dec80bbcc\": container with ID starting with ac46986bdef3856072a0ab53c7e690a75d9755efb227605139d3ea1dec80bbcc not found: ID does not exist" containerID="ac46986bdef3856072a0ab53c7e690a75d9755efb227605139d3ea1dec80bbcc"
Nov 02 10:01:24.053059 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.052991    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac46986bdef3856072a0ab53c7e690a75d9755efb227605139d3ea1dec80bbcc"} err="failed to get container status \"ac46986bdef3856072a0ab53c7e690a75d9755efb227605139d3ea1dec80bbcc\": rpc error: code = NotFound desc = could not find container \"ac46986bdef3856072a0ab53c7e690a75d9755efb227605139d3ea1dec80bbcc\": container with ID starting with ac46986bdef3856072a0ab53c7e690a75d9755efb227605139d3ea1dec80bbcc not found: ID does not exist"
Nov 02 10:01:24.064026 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.064001    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"]
Nov 02 10:01:24.068885 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.068860    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx"]
Nov 02 10:01:24.135966 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.135883    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:24.163586 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.163551    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh" podUID="6d32044d-b33f-4d9d-8e4a-c974bb485074" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:01:24.425487 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:01:24.425453    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5e9598_1b27_47e1_ae1b_81ca30d1bcc8.slice/crio-db3c7e56306c4b21ea87a48cee13ce3bd3a5324251258045ff9f355cd9c38d53 WatchSource:0}: Error finding container db3c7e56306c4b21ea87a48cee13ce3bd3a5324251258045ff9f355cd9c38d53: Status 404 returned error can't find the container with id db3c7e56306c4b21ea87a48cee13ce3bd3a5324251258045ff9f355cd9c38d53
Nov 02 10:01:24.425812 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.425528    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"]
Nov 02 10:01:24.427801 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.427768    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:01:24.427896 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.427831    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:01:24.427896 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.427860    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:01:24.489042 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:24.489013    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-2-w3-a110ef-kserve-86cc885b6f-9h5mx" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 02 10:01:25.066687 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:25.056563    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r" event={"ID":"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8","Type":"ContainerStarted","Data":"1387a925602f7c18f2f30775b49d637f3d0f3800b30291d3fba9e9a62afcf0c6"}
Nov 02 10:01:25.066687 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:25.056649    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r" event={"ID":"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8","Type":"ContainerStarted","Data":"db3c7e56306c4b21ea87a48cee13ce3bd3a5324251258045ff9f355cd9c38d53"}
Nov 02 10:01:25.096533 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:25.096477    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r" podStartSLOduration=2.096460961 podStartE2EDuration="2.096460961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:01:25.092972812 +0000 UTC m=+1562.031259607" watchObservedRunningTime="2026-11-02 10:01:25.096460961 +0000 UTC m=+1562.034747795"
Nov 02 10:01:25.136357 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:25.136324    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:25.137770 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:25.137747    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 10:01:25.137901 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:25.137788    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r" podUID="7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 10:01:25.206006 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:25.205975    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7251496-4b91-4330-9eb1-f6c91492f2aa" path="/var/lib/kubelet/pods/b7251496-4b91-4330-9eb1-f6c91492f2aa/volumes"
Nov 02 10:01:25.206415 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:25.206401    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3033ba-0513-428c-9380-50d8f9f3bca8" path="/var/lib/kubelet/pods/cb3033ba-0513-428c-9380-50d8f9f3bca8/volumes"
Nov 02 10:01:26.136617 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:26.136581    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 10:01:26.137122 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:26.136639    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r" podUID="7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 10:01:27.136993 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:27.136949    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 10:01:27.137454 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:27.137015    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r" podUID="7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 10:01:28.136492 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:28.136456    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 10:01:28.136685 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:28.136513    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r" podUID="7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 10:01:29.136624 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:29.136585    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 10:01:29.137013 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:29.136642    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r" podUID="7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 10:01:30.136641 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:30.136601    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 10:01:30.137053 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:30.136662    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r" podUID="7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 10:01:31.136815 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:31.136770    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 10:01:31.137252 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:31.136859    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r" podUID="7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 10:01:32.589797 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:32.589759    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:32.590548 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:32.590510    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:32.591473 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:32.591452    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:34.165568 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:34.165525    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh" podUID="6d32044d-b33f-4d9d-8e4a-c974bb485074" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:01:36.551946 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:01:36.551912    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:01:40.688438 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:40.688406    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"
Nov 02 10:01:40.814797 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:40.814760    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d32044d-b33f-4d9d-8e4a-c974bb485074-kube-api-access-5wr28\" (UniqueName: \"kubernetes.io/projected/6d32044d-b33f-4d9d-8e4a-c974bb485074-kube-api-access-5wr28\") pod \"6d32044d-b33f-4d9d-8e4a-c974bb485074\" (UID: \"6d32044d-b33f-4d9d-8e4a-c974bb485074\") "
Nov 02 10:01:40.815019 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:40.814962    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d32044d-b33f-4d9d-8e4a-c974bb485074-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d32044d-b33f-4d9d-8e4a-c974bb485074-plugins-config-volume\") pod \"6d32044d-b33f-4d9d-8e4a-c974bb485074\" (UID: \"6d32044d-b33f-4d9d-8e4a-c974bb485074\") "
Nov 02 10:01:40.815381 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:40.815354    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d32044d-b33f-4d9d-8e4a-c974bb485074-plugins-config-volume" pod "6d32044d-b33f-4d9d-8e4a-c974bb485074" (UID: "6d32044d-b33f-4d9d-8e4a-c974bb485074"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:01:40.817267 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:40.817238    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d32044d-b33f-4d9d-8e4a-c974bb485074-kube-api-access-5wr28" pod "6d32044d-b33f-4d9d-8e4a-c974bb485074" (UID: "6d32044d-b33f-4d9d-8e4a-c974bb485074"). InnerVolumeSpecName "kube-api-access-5wr28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:40.916707 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:40.916620    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wr28\" (UniqueName: \"kubernetes.io/projected/6d32044d-b33f-4d9d-8e4a-c974bb485074-kube-api-access-5wr28\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:40.916707 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:40.916655    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d32044d-b33f-4d9d-8e4a-c974bb485074-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:41.225913 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:41.225881    4449 generic.go:356] "Generic (PLEG): container finished" podID="6d32044d-b33f-4d9d-8e4a-c974bb485074" containerID="5df174620b4cd75f10c19d06a5db9d97cd3a5bcde6098dd82d111552c5f6ea56" exitCode=0
Nov 02 10:01:41.226066 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:41.225938    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"
Nov 02 10:01:41.226066 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:41.225961    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh" event={"ID":"6d32044d-b33f-4d9d-8e4a-c974bb485074","Type":"ContainerDied","Data":"5df174620b4cd75f10c19d06a5db9d97cd3a5bcde6098dd82d111552c5f6ea56"}
Nov 02 10:01:41.226066 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:41.226017    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh" event={"ID":"6d32044d-b33f-4d9d-8e4a-c974bb485074","Type":"ContainerDied","Data":"24c0624288d81219abdde050dc0696089c5d20faf65a1c0de13c82bcd467bda2"}
Nov 02 10:01:41.226066 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:41.226057    4449 scope.go:122] "RemoveContainer" containerID="5df174620b4cd75f10c19d06a5db9d97cd3a5bcde6098dd82d111552c5f6ea56"
Nov 02 10:01:41.236397 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:41.236376    4449 scope.go:122] "RemoveContainer" containerID="5df174620b4cd75f10c19d06a5db9d97cd3a5bcde6098dd82d111552c5f6ea56"
Nov 02 10:01:41.236669 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:01:41.236646    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df174620b4cd75f10c19d06a5db9d97cd3a5bcde6098dd82d111552c5f6ea56\": container with ID starting with 5df174620b4cd75f10c19d06a5db9d97cd3a5bcde6098dd82d111552c5f6ea56 not found: ID does not exist" containerID="5df174620b4cd75f10c19d06a5db9d97cd3a5bcde6098dd82d111552c5f6ea56"
Nov 02 10:01:41.236728 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:41.236681    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df174620b4cd75f10c19d06a5db9d97cd3a5bcde6098dd82d111552c5f6ea56"} err="failed to get container status \"5df174620b4cd75f10c19d06a5db9d97cd3a5bcde6098dd82d111552c5f6ea56\": rpc error: code = NotFound desc = could not find container \"5df174620b4cd75f10c19d06a5db9d97cd3a5bcde6098dd82d111552c5f6ea56\": container with ID starting with 5df174620b4cd75f10c19d06a5db9d97cd3a5bcde6098dd82d111552c5f6ea56 not found: ID does not exist"
Nov 02 10:01:41.246536 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:41.246514    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"]
Nov 02 10:01:41.249876 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:41.249855    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a110ef-c6fc947bc-hx6lh"]
Nov 02 10:01:43.206427 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:43.206391    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d32044d-b33f-4d9d-8e4a-c974bb485074" path="/var/lib/kubelet/pods/6d32044d-b33f-4d9d-8e4a-c974bb485074/volumes"
Nov 02 10:01:50.486116 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:50.486068    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"]
Nov 02 10:01:50.486637 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:50.486351    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r" podUID="7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" containerName="istio-proxy" containerID="cri-o://1387a925602f7c18f2f30775b49d637f3d0f3800b30291d3fba9e9a62afcf0c6" gracePeriod=30
Nov 02 10:01:50.491599 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:50.491571    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"]
Nov 02 10:01:50.491799 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:50.491764    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" podUID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" containerName="istio-proxy" containerID="cri-o://cc823ff680fc5ec8259e6d3c1d6bd3e5fc32ebdc45e72b5973902af52dc77b2c" gracePeriod=30
Nov 02 10:01:50.494865 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:50.494843    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"]
Nov 02 10:01:50.495044 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:50.495010    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy" containerID="cri-o://770df8433dab0e28bd44ad297b91bb41c46174a96da4c9e6cae3dbed5519542d" gracePeriod=30
Nov 02 10:01:53.581365 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:53.581326    4449 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 10:01:53.581750 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:53.581436    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 10:01:55.784035 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.783961    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:55.841449 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.841422    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-token\" (UniqueName: \"kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-token\") pod \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") "
Nov 02 10:01:55.841589 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.841459    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-kube-api-access-zknpv\" (UniqueName: \"kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-kube-api-access-zknpv\") pod \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") "
Nov 02 10:01:55.841589 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.841558    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-envoy\") pod \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") "
Nov 02 10:01:55.841589 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.841584    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istiod-ca-cert\") pod \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") "
Nov 02 10:01:55.841766 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.841634    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-socket\") pod \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") "
Nov 02 10:01:55.841766 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.841653    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-certs\") pod \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") "
Nov 02 10:01:55.841766 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.841706    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-credential-socket\") pod \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") "
Nov 02 10:01:55.841766 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.841731    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-podinfo\") pod \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") "
Nov 02 10:01:55.841766 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.841747    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-data\") pod \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\" (UID: \"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8\") "
Nov 02 10:01:55.842764 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.842654    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istiod-ca-cert" pod "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" (UID: "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:01:55.842889 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.842708    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-certs" pod "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" (UID: "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.843104 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.843079    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-data" pod "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" (UID: "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.843343 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.843303    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-socket" pod "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" (UID: "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.843703 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.843680    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-credential-socket" pod "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" (UID: "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.845777 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.845746    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-envoy" pod "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" (UID: "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.847681 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.847554    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-kube-api-access-zknpv" pod "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" (UID: "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8"). InnerVolumeSpecName "kube-api-access-zknpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:55.847681 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.847582    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-token" pod "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" (UID: "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:55.848958 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.848932    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-podinfo" pod "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" (UID: "7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:01:55.849983 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.849963    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 10:01:55.853658 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.853637    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 10:01:55.942989 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.942954    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-credential-socket\") pod \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") "
Nov 02 10:01:55.943181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943000    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d365185-6209-4973-8a3f-52e66fe3181f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d365185-6209-4973-8a3f-52e66fe3181f-istio-podinfo\") pod \"8d365185-6209-4973-8a3f-52e66fe3181f\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") "
Nov 02 10:01:55.943181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943025    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-data\") pod \"8d365185-6209-4973-8a3f-52e66fe3181f\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") "
Nov 02 10:01:55.943181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943047    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-kube-api-access-29bf5\" (UniqueName: \"kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-kube-api-access-29bf5\") pod \"8d365185-6209-4973-8a3f-52e66fe3181f\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") "
Nov 02 10:01:55.943181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943073    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istiod-ca-cert\") pod \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") "
Nov 02 10:01:55.943181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943099    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-envoy\") pod \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") "
Nov 02 10:01:55.943528 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943200    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-envoy\") pod \"8d365185-6209-4973-8a3f-52e66fe3181f\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") "
Nov 02 10:01:55.943528 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943281    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-credential-socket" pod "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" (UID: "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.943528 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943296    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-podinfo\") pod \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") "
Nov 02 10:01:55.943528 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943412    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-data\") pod \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") "
Nov 02 10:01:55.943528 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943471    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-token\" (UniqueName: \"kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-token\") pod \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") "
Nov 02 10:01:55.943528 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943504    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-certs\") pod \"8d365185-6209-4973-8a3f-52e66fe3181f\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") "
Nov 02 10:01:55.943934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943547    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-istio-token\" (UniqueName: \"kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-istio-token\") pod \"8d365185-6209-4973-8a3f-52e66fe3181f\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") "
Nov 02 10:01:55.943934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943411    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-data" pod "8d365185-6209-4973-8a3f-52e66fe3181f" (UID: "8d365185-6209-4973-8a3f-52e66fe3181f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.943934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943565    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istiod-ca-cert" pod "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" (UID: "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:01:55.943934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943623    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-certs\") pod \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") "
Nov 02 10:01:55.943934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943650    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-kube-api-access-4x7f2\" (UniqueName: \"kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-kube-api-access-4x7f2\") pod \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") "
Nov 02 10:01:55.943934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943710    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-socket\") pod \"8d365185-6209-4973-8a3f-52e66fe3181f\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") "
Nov 02 10:01:55.943934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943744    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d365185-6209-4973-8a3f-52e66fe3181f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d365185-6209-4973-8a3f-52e66fe3181f-istiod-ca-cert\") pod \"8d365185-6209-4973-8a3f-52e66fe3181f\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") "
Nov 02 10:01:55.943934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943788    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-credential-socket\") pod \"8d365185-6209-4973-8a3f-52e66fe3181f\" (UID: \"8d365185-6209-4973-8a3f-52e66fe3181f\") "
Nov 02 10:01:55.943934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943820    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-socket\") pod \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\" (UID: \"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b\") "
Nov 02 10:01:55.943934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.943869    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-data" pod "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" (UID: "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.944567 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944067    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-certs" pod "8d365185-6209-4973-8a3f-52e66fe3181f" (UID: "8d365185-6209-4973-8a3f-52e66fe3181f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.944567 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944242    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-certs" pod "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" (UID: "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.944763 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944627    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.944763 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944650    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.944763 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944665    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.944763 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944680    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.944763 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944695    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.944763 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944708    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.944763 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944734    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.944763 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944748    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.944763 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944761    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.944763 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944775    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zknpv\" (UniqueName: \"kubernetes.io/projected/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-kube-api-access-zknpv\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.944763 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944788    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.944763 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944802    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.945541 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944817    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.945541 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944831    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.945541 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.944844    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:55.946305 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.946274    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-kube-api-access-29bf5" pod "8d365185-6209-4973-8a3f-52e66fe3181f" (UID: "8d365185-6209-4973-8a3f-52e66fe3181f"). InnerVolumeSpecName "kube-api-access-29bf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:55.946993 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.946543    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d365185-6209-4973-8a3f-52e66fe3181f-istiod-ca-cert" pod "8d365185-6209-4973-8a3f-52e66fe3181f" (UID: "8d365185-6209-4973-8a3f-52e66fe3181f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:01:55.946993 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.946789    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-socket" pod "8d365185-6209-4973-8a3f-52e66fe3181f" (UID: "8d365185-6209-4973-8a3f-52e66fe3181f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.946993 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.946892    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-envoy" pod "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" (UID: "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.946993 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.946937    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d365185-6209-4973-8a3f-52e66fe3181f-istio-podinfo" pod "8d365185-6209-4973-8a3f-52e66fe3181f" (UID: "8d365185-6209-4973-8a3f-52e66fe3181f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:01:55.946993 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.946966    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-credential-socket" pod "8d365185-6209-4973-8a3f-52e66fe3181f" (UID: "8d365185-6209-4973-8a3f-52e66fe3181f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.946993 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.946972    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-istio-token" pod "8d365185-6209-4973-8a3f-52e66fe3181f" (UID: "8d365185-6209-4973-8a3f-52e66fe3181f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:55.947464 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.947155    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-socket" pod "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" (UID: "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.947464 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.947248    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-token" pod "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" (UID: "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:55.947464 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.947427    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-podinfo" pod "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" (UID: "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:01:55.947571 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.947468    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-kube-api-access-4x7f2" pod "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" (UID: "0c0663f6-24d9-42e1-b69d-d1c73b24fa6b"). InnerVolumeSpecName "kube-api-access-4x7f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:55.947571 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:55.947546    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-envoy" pod "8d365185-6209-4973-8a3f-52e66fe3181f" (UID: "8d365185-6209-4973-8a3f-52e66fe3181f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:56.045607 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.045575    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.045607 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.045604    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d365185-6209-4973-8a3f-52e66fe3181f-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.045813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.045616    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.045813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.045627    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.045813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.045635    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d365185-6209-4973-8a3f-52e66fe3181f-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.045813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.045643    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29bf5\" (UniqueName: \"kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-kube-api-access-29bf5\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.045813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.045654    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.045813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.045662    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d365185-6209-4973-8a3f-52e66fe3181f-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.045813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.045670    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.045813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.045678    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.045813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.045686    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d365185-6209-4973-8a3f-52e66fe3181f-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.045813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.045693    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x7f2\" (UniqueName: \"kubernetes.io/projected/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b-kube-api-access-4x7f2\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.350669 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.350573    4449 generic.go:356] "Generic (PLEG): container finished" podID="8d365185-6209-4973-8a3f-52e66fe3181f" containerID="770df8433dab0e28bd44ad297b91bb41c46174a96da4c9e6cae3dbed5519542d" exitCode=0
Nov 02 10:01:56.350669 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.350631    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"
Nov 02 10:01:56.350669 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.350653    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" event={"ID":"8d365185-6209-4973-8a3f-52e66fe3181f","Type":"ContainerDied","Data":"770df8433dab0e28bd44ad297b91bb41c46174a96da4c9e6cae3dbed5519542d"}
Nov 02 10:01:56.350962 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.350701    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg" event={"ID":"8d365185-6209-4973-8a3f-52e66fe3181f","Type":"ContainerDied","Data":"d67eddfccff727b8dc363c4963a7e009a279255de663eca91e73cdbdab5773e8"}
Nov 02 10:01:56.350962 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.350730    4449 scope.go:122] "RemoveContainer" containerID="770df8433dab0e28bd44ad297b91bb41c46174a96da4c9e6cae3dbed5519542d"
Nov 02 10:01:56.358256 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.352420    4449 generic.go:356] "Generic (PLEG): container finished" podID="7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" containerID="1387a925602f7c18f2f30775b49d637f3d0f3800b30291d3fba9e9a62afcf0c6" exitCode=0
Nov 02 10:01:56.358256 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.352503    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"
Nov 02 10:01:56.358256 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.352534    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r" event={"ID":"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8","Type":"ContainerDied","Data":"1387a925602f7c18f2f30775b49d637f3d0f3800b30291d3fba9e9a62afcf0c6"}
Nov 02 10:01:56.358256 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.352589    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r" event={"ID":"7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8","Type":"ContainerDied","Data":"db3c7e56306c4b21ea87a48cee13ce3bd3a5324251258045ff9f355cd9c38d53"}
Nov 02 10:01:56.358256 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.354179    4449 generic.go:356] "Generic (PLEG): container finished" podID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" containerID="cc823ff680fc5ec8259e6d3c1d6bd3e5fc32ebdc45e72b5973902af52dc77b2c" exitCode=0
Nov 02 10:01:56.358256 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.354268    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"
Nov 02 10:01:56.358256 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.354252    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" event={"ID":"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b","Type":"ContainerDied","Data":"cc823ff680fc5ec8259e6d3c1d6bd3e5fc32ebdc45e72b5973902af52dc77b2c"}
Nov 02 10:01:56.358256 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.354320    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5" event={"ID":"0c0663f6-24d9-42e1-b69d-d1c73b24fa6b","Type":"ContainerDied","Data":"16d7fd1b833348ea1de70cd32dbd22adaf8b6d5ac0b084767552d0ccc39b4239"}
Nov 02 10:01:56.362638 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.362621    4449 scope.go:122] "RemoveContainer" containerID="b5df30b3cc25eefccc19855a3ce5d91fb990e552d8ab9ee32bd0bb5ade864ac0"
Nov 02 10:01:56.372666 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.372649    4449 scope.go:122] "RemoveContainer" containerID="770df8433dab0e28bd44ad297b91bb41c46174a96da4c9e6cae3dbed5519542d"
Nov 02 10:01:56.372945 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:01:56.372925    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770df8433dab0e28bd44ad297b91bb41c46174a96da4c9e6cae3dbed5519542d\": container with ID starting with 770df8433dab0e28bd44ad297b91bb41c46174a96da4c9e6cae3dbed5519542d not found: ID does not exist" containerID="770df8433dab0e28bd44ad297b91bb41c46174a96da4c9e6cae3dbed5519542d"
Nov 02 10:01:56.373029 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.372959    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770df8433dab0e28bd44ad297b91bb41c46174a96da4c9e6cae3dbed5519542d"} err="failed to get container status \"770df8433dab0e28bd44ad297b91bb41c46174a96da4c9e6cae3dbed5519542d\": rpc error: code = NotFound desc = could not find container \"770df8433dab0e28bd44ad297b91bb41c46174a96da4c9e6cae3dbed5519542d\": container with ID starting with 770df8433dab0e28bd44ad297b91bb41c46174a96da4c9e6cae3dbed5519542d not found: ID does not exist"
Nov 02 10:01:56.373029 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.372984    4449 scope.go:122] "RemoveContainer" containerID="b5df30b3cc25eefccc19855a3ce5d91fb990e552d8ab9ee32bd0bb5ade864ac0"
Nov 02 10:01:56.373283 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:01:56.373266    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5df30b3cc25eefccc19855a3ce5d91fb990e552d8ab9ee32bd0bb5ade864ac0\": container with ID starting with b5df30b3cc25eefccc19855a3ce5d91fb990e552d8ab9ee32bd0bb5ade864ac0 not found: ID does not exist" containerID="b5df30b3cc25eefccc19855a3ce5d91fb990e552d8ab9ee32bd0bb5ade864ac0"
Nov 02 10:01:56.373333 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.373292    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5df30b3cc25eefccc19855a3ce5d91fb990e552d8ab9ee32bd0bb5ade864ac0"} err="failed to get container status \"b5df30b3cc25eefccc19855a3ce5d91fb990e552d8ab9ee32bd0bb5ade864ac0\": rpc error: code = NotFound desc = could not find container \"b5df30b3cc25eefccc19855a3ce5d91fb990e552d8ab9ee32bd0bb5ade864ac0\": container with ID starting with b5df30b3cc25eefccc19855a3ce5d91fb990e552d8ab9ee32bd0bb5ade864ac0 not found: ID does not exist"
Nov 02 10:01:56.373333 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.373308    4449 scope.go:122] "RemoveContainer" containerID="1387a925602f7c18f2f30775b49d637f3d0f3800b30291d3fba9e9a62afcf0c6"
Nov 02 10:01:56.383613 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.383590    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"]
Nov 02 10:01:56.385666 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.385647    4449 scope.go:122] "RemoveContainer" containerID="1387a925602f7c18f2f30775b49d637f3d0f3800b30291d3fba9e9a62afcf0c6"
Nov 02 10:01:56.385969 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:01:56.385948    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1387a925602f7c18f2f30775b49d637f3d0f3800b30291d3fba9e9a62afcf0c6\": container with ID starting with 1387a925602f7c18f2f30775b49d637f3d0f3800b30291d3fba9e9a62afcf0c6 not found: ID does not exist" containerID="1387a925602f7c18f2f30775b49d637f3d0f3800b30291d3fba9e9a62afcf0c6"
Nov 02 10:01:56.386039 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.385976    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1387a925602f7c18f2f30775b49d637f3d0f3800b30291d3fba9e9a62afcf0c6"} err="failed to get container status \"1387a925602f7c18f2f30775b49d637f3d0f3800b30291d3fba9e9a62afcf0c6\": rpc error: code = NotFound desc = could not find container \"1387a925602f7c18f2f30775b49d637f3d0f3800b30291d3fba9e9a62afcf0c6\": container with ID starting with 1387a925602f7c18f2f30775b49d637f3d0f3800b30291d3fba9e9a62afcf0c6 not found: ID does not exist"
Nov 02 10:01:56.386039 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.385993    4449 scope.go:122] "RemoveContainer" containerID="cc823ff680fc5ec8259e6d3c1d6bd3e5fc32ebdc45e72b5973902af52dc77b2c"
Nov 02 10:01:56.388882 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.388862    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-m8mhg"]
Nov 02 10:01:56.395807 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.395786    4449 scope.go:122] "RemoveContainer" containerID="cc823ff680fc5ec8259e6d3c1d6bd3e5fc32ebdc45e72b5973902af52dc77b2c"
Nov 02 10:01:56.396121 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:01:56.396097    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc823ff680fc5ec8259e6d3c1d6bd3e5fc32ebdc45e72b5973902af52dc77b2c\": container with ID starting with cc823ff680fc5ec8259e6d3c1d6bd3e5fc32ebdc45e72b5973902af52dc77b2c not found: ID does not exist" containerID="cc823ff680fc5ec8259e6d3c1d6bd3e5fc32ebdc45e72b5973902af52dc77b2c"
Nov 02 10:01:56.396187 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.396124    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc823ff680fc5ec8259e6d3c1d6bd3e5fc32ebdc45e72b5973902af52dc77b2c"} err="failed to get container status \"cc823ff680fc5ec8259e6d3c1d6bd3e5fc32ebdc45e72b5973902af52dc77b2c\": rpc error: code = NotFound desc = could not find container \"cc823ff680fc5ec8259e6d3c1d6bd3e5fc32ebdc45e72b5973902af52dc77b2c\": container with ID starting with cc823ff680fc5ec8259e6d3c1d6bd3e5fc32ebdc45e72b5973902af52dc77b2c not found: ID does not exist"
Nov 02 10:01:56.400535 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.400510    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"]
Nov 02 10:01:56.407434 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.407414    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2rh7r"]
Nov 02 10:01:56.418353 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.418333    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"]
Nov 02 10:01:56.423048 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:56.423027    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-p5zd5"]
Nov 02 10:01:57.205823 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:57.205786    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" path="/var/lib/kubelet/pods/0c0663f6-24d9-42e1-b69d-d1c73b24fa6b/volumes"
Nov 02 10:01:57.206298 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:57.206283    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" path="/var/lib/kubelet/pods/7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8/volumes"
Nov 02 10:01:57.206705 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:01:57.206692    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" path="/var/lib/kubelet/pods/8d365185-6209-4973-8a3f-52e66fe3181f/volumes"
Nov 02 10:02:06.548520 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:02:06.548482    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:02:24.031133 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:02:24.031101    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"]
Nov 02 10:02:24.034288 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:02:24.034262    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d9c5a2-ghl8g"]
Nov 02 10:02:25.205958 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:02:25.205920    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3766ce9-16a9-41e0-adf3-8871e5bd8bc0" path="/var/lib/kubelet/pods/f3766ce9-16a9-41e0-adf3-8871e5bd8bc0/volumes"
Nov 02 10:02:25.665610 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:02:25.665580    4449 scope.go:122] "RemoveContainer" containerID="1977f75ce70c98001bf1d50fb2e160bf8ff3157cbb9ea29cfd1440ba6185901a"
Nov 02 10:02:25.675441 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:02:25.675418    4449 scope.go:122] "RemoveContainer" containerID="09ab49d976642b9d4547bba9f8a89a06b42b8da1e7e9b7f89e772647ed2b56aa"
Nov 02 10:02:25.685524 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:02:25.685498    4449 scope.go:122] "RemoveContainer" containerID="327c01f437500bd987058811f0fac954fd7103dd10f4b5901ecad361d6f23a20"
Nov 02 10:02:25.694498 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:02:25.694477    4449 scope.go:122] "RemoveContainer" containerID="d6110df14fabb78a03a2be33a21423a8d190b96f58ee8a378dceda39c8d68543"
Nov 02 10:02:25.703138 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:02:25.703119    4449 scope.go:122] "RemoveContainer" containerID="64c5b65c1481a3aea072e7933b53392f8cabf760aab8e473f38d49d1bdedbbdb"
Nov 02 10:02:25.711447 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:02:25.711431    4449 scope.go:122] "RemoveContainer" containerID="49cb4eeeb605f47ba8a63fa42196d552592381b596a740b6d58c065c78752ad2"
Nov 02 10:02:25.721792 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:02:25.721772    4449 scope.go:122] "RemoveContainer" containerID="e4f76e78cc47bdaae3856dcb93fe5b48fba573e60ba6d0f1d064206afe50fcf3"
Nov 02 10:02:36.547071 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:02:36.547038    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:03:02.031932 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:02.031893    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"]
Nov 02 10:03:02.036032 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:02.036000    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"]
Nov 02 10:03:02.038891 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:02.038870    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc4208e-79hx9"]
Nov 02 10:03:02.041668 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:02.041642    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0fd74e2-8tmxg"]
Nov 02 10:03:03.211030 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:03.210990    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="640c1730-9543-4fce-b997-56ca87b67dfe" path="/var/lib/kubelet/pods/640c1730-9543-4fce-b997-56ca87b67dfe/volumes"
Nov 02 10:03:03.211475 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:03.211460    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd" path="/var/lib/kubelet/pods/cd14e18c-0c04-4e1d-90b2-c935c7e9cdfd/volumes"
Nov 02 10:03:06.548138 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:03:06.548039    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:03:14.026475 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:14.026435    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"]
Nov 02 10:03:14.029639 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:14.029613    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f8ebf2-qdgl8"]
Nov 02 10:03:15.026773 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:15.026737    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"]
Nov 02 10:03:15.029692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:15.029665    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775343e0-vxqj7"]
Nov 02 10:03:15.207913 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:15.207811    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac8f37f0-1a7c-404b-a032-c7b221388227" path="/var/lib/kubelet/pods/ac8f37f0-1a7c-404b-a032-c7b221388227/volumes"
Nov 02 10:03:15.208378 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:15.208355    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23a5653-492b-4f80-b6ca-f32b35c49582" path="/var/lib/kubelet/pods/e23a5653-492b-4f80-b6ca-f32b35c49582/volumes"
Nov 02 10:03:18.459943 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.459908    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"]
Nov 02 10:03:18.460472 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460440    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" containerName="istio-proxy"
Nov 02 10:03:18.460472 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460452    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" containerName="istio-proxy"
Nov 02 10:03:18.460472 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460472    4449 cpu_manager.go:414] "containerMap: removing container" podUID="7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" containerName="istio-proxy"
Nov 02 10:03:18.460600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460477    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" containerName="istio-proxy"
Nov 02 10:03:18.460600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460486    4449 cpu_manager.go:414] "containerMap: removing container" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy"
Nov 02 10:03:18.460600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460493    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy"
Nov 02 10:03:18.460600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460510    4449 cpu_manager.go:414] "containerMap: removing container" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy"
Nov 02 10:03:18.460600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460515    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy"
Nov 02 10:03:18.460600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460528    4449 cpu_manager.go:414] "containerMap: removing container" podUID="6d32044d-b33f-4d9d-8e4a-c974bb485074" containerName="payload-pre-processing"
Nov 02 10:03:18.460600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460532    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d32044d-b33f-4d9d-8e4a-c974bb485074" containerName="payload-pre-processing"
Nov 02 10:03:18.460820 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460609    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0663f6-24d9-42e1-b69d-d1c73b24fa6b" containerName="istio-proxy"
Nov 02 10:03:18.460820 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460619    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5e9598-1b27-47e1-ae1b-81ca30d1bcc8" containerName="istio-proxy"
Nov 02 10:03:18.460820 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460628    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy"
Nov 02 10:03:18.460820 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460634    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d365185-6209-4973-8a3f-52e66fe3181f" containerName="istio-proxy"
Nov 02 10:03:18.460820 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.460641    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d32044d-b33f-4d9d-8e4a-c974bb485074" containerName="payload-pre-processing"
Nov 02 10:03:18.464099 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.464079    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"
Nov 02 10:03:18.469518 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.469007    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:03:18.470792 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.470605    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:03:18.470792 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.470721    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 10:03:18.470792 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.470755    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:03:18.477117 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.477085    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"]
Nov 02 10:03:18.616954 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.616914    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1a4f6f-stbts\" (UID: \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"
Nov 02 10:03:18.617147 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.616967    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1a4f6f-stbts\" (UID: \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"
Nov 02 10:03:18.617147 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.617084    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7qb7\" (UniqueName: \"kubernetes.io/projected/fea54a6f-c7ec-4a58-8d50-205640bf1e31-kube-api-access-f7qb7\") pod \"maas-api-revoke-sub-0f1a4f6f-stbts\" (UID: \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"
Nov 02 10:03:18.718537 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.718448    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7qb7\" (UniqueName: \"kubernetes.io/projected/fea54a6f-c7ec-4a58-8d50-205640bf1e31-kube-api-access-f7qb7\") pod \"maas-api-revoke-sub-0f1a4f6f-stbts\" (UID: \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"
Nov 02 10:03:18.718714 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.718545    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1a4f6f-stbts\" (UID: \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"
Nov 02 10:03:18.718714 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.718577    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1a4f6f-stbts\" (UID: \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"
Nov 02 10:03:18.719314 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.719285    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1a4f6f-stbts\" (UID: \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"
Nov 02 10:03:18.721140 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.721113    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1a4f6f-stbts\" (UID: \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"
Nov 02 10:03:18.736198 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.736161    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7qb7\" (UniqueName: \"kubernetes.io/projected/fea54a6f-c7ec-4a58-8d50-205640bf1e31-kube-api-access-f7qb7\") pod \"maas-api-revoke-sub-0f1a4f6f-stbts\" (UID: \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"
Nov 02 10:03:18.794007 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:18.793975    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"
Nov 02 10:03:19.094943 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.094912    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"]
Nov 02 10:03:19.095720 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:03:19.095683    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea54a6f_c7ec_4a58_8d50_205640bf1e31.slice/crio-f21b63049eb5f7b16e581840e12f51386b969b5096781352ad34bca0ce0f3e53 WatchSource:0}: Error finding container f21b63049eb5f7b16e581840e12f51386b969b5096781352ad34bca0ce0f3e53: Status 404 returned error can't find the container with id f21b63049eb5f7b16e581840e12f51386b969b5096781352ad34bca0ce0f3e53
Nov 02 10:03:19.406619 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.406582    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"]
Nov 02 10:03:19.412102 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.412069    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.423534 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.423498    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"]
Nov 02 10:03:19.539415 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.539370    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37648fb-1120-4f9d-a25b-2158bbbd3483-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.539971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.539437    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.539971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.539510    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.539971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.539542    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.539971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.539636    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.539971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.539695    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.539971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.539763    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.539971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.539793    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.539971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.539896    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsxpx\" (UniqueName: \"kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-kube-api-access-dsxpx\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.640790 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.640691    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.640790 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.640759    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.641039 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.640820    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.641039 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.640938    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.641164 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.641045    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsxpx\" (UniqueName: \"kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-kube-api-access-dsxpx\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.641164 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.641074    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37648fb-1120-4f9d-a25b-2158bbbd3483-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.641164 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.641092    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.641414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.641276    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.641414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.641307    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.641414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.641317    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.641414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.641350    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.641414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.641390    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.641657 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.641585    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.642115 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.642089    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37648fb-1120-4f9d-a25b-2158bbbd3483-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.643435 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.643410    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.643618 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.643599    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.653561 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.653541    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.653891 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.653867    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsxpx\" (UniqueName: \"kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-kube-api-access-dsxpx\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.734296 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.734261    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:19.990196 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.990165    4449 generic.go:356] "Generic (PLEG): container finished" podID="fea54a6f-c7ec-4a58-8d50-205640bf1e31" containerID="73f2f008e2aec46919f7e9b9f8519795ee2cfbb40d4bcfef8704c6a4cc1f5736" exitCode=0
Nov 02 10:03:19.990382 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.990259    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts" event={"ID":"fea54a6f-c7ec-4a58-8d50-205640bf1e31","Type":"ContainerDied","Data":"73f2f008e2aec46919f7e9b9f8519795ee2cfbb40d4bcfef8704c6a4cc1f5736"}
Nov 02 10:03:19.990382 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:19.990331    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts" event={"ID":"fea54a6f-c7ec-4a58-8d50-205640bf1e31","Type":"ContainerStarted","Data":"f21b63049eb5f7b16e581840e12f51386b969b5096781352ad34bca0ce0f3e53"}
Nov 02 10:03:20.025518 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:20.025490    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"]
Nov 02 10:03:20.034427 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:03:20.034395    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37648fb_1120_4f9d_a25b_2158bbbd3483.slice/crio-701ec9a71a62db5e28e8f11bd34ac78e6e1480d108210516d37a04d0c3a9df18 WatchSource:0}: Error finding container 701ec9a71a62db5e28e8f11bd34ac78e6e1480d108210516d37a04d0c3a9df18: Status 404 returned error can't find the container with id 701ec9a71a62db5e28e8f11bd34ac78e6e1480d108210516d37a04d0c3a9df18
Nov 02 10:03:20.036835 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:20.036801    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:03:20.036940 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:20.036882    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:03:20.036940 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:20.036929    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:03:21.001590 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.001557    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx" event={"ID":"a37648fb-1120-4f9d-a25b-2158bbbd3483","Type":"ContainerStarted","Data":"0a9f0a8a41322ef199e57fcd5a42b9caf424125eda136439c2e60919a95370b6"}
Nov 02 10:03:21.002029 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.001610    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx" event={"ID":"a37648fb-1120-4f9d-a25b-2158bbbd3483","Type":"ContainerStarted","Data":"701ec9a71a62db5e28e8f11bd34ac78e6e1480d108210516d37a04d0c3a9df18"}
Nov 02 10:03:21.028180 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.028120    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx" podStartSLOduration=2.028103768 podStartE2EDuration="2.028103768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:03:21.026078964 +0000 UTC m=+1677.964365740" watchObservedRunningTime="2026-11-02 10:03:21.028103768 +0000 UTC m=+1677.966390544"
Nov 02 10:03:21.215284 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.215258    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"
Nov 02 10:03:21.281045 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.280937    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-openshift-ca\") pod \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\" (UID: \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\") "
Nov 02 10:03:21.281258 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.281101    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-xks-ca\") pod \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\" (UID: \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\") "
Nov 02 10:03:21.281258 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.281205    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea54a6f-c7ec-4a58-8d50-205640bf1e31-kube-api-access-f7qb7\" (UniqueName: \"kubernetes.io/projected/fea54a6f-c7ec-4a58-8d50-205640bf1e31-kube-api-access-f7qb7\") pod \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\" (UID: \"fea54a6f-c7ec-4a58-8d50-205640bf1e31\") "
Nov 02 10:03:21.281762 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.281729    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-openshift-ca" pod "fea54a6f-c7ec-4a58-8d50-205640bf1e31" (UID: "fea54a6f-c7ec-4a58-8d50-205640bf1e31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:21.283744 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.283708    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea54a6f-c7ec-4a58-8d50-205640bf1e31-kube-api-access-f7qb7" pod "fea54a6f-c7ec-4a58-8d50-205640bf1e31" (UID: "fea54a6f-c7ec-4a58-8d50-205640bf1e31"). InnerVolumeSpecName "kube-api-access-f7qb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:21.283976 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.283955    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-xks-ca" pod "fea54a6f-c7ec-4a58-8d50-205640bf1e31" (UID: "fea54a6f-c7ec-4a58-8d50-205640bf1e31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:03:21.382459 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.382420    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:21.382459 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.382454    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea54a6f-c7ec-4a58-8d50-205640bf1e31-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:21.382692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.382470    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7qb7\" (UniqueName: \"kubernetes.io/projected/fea54a6f-c7ec-4a58-8d50-205640bf1e31-kube-api-access-f7qb7\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:21.735199 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.735153    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:21.736695 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.736669    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 10:03:21.736840 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:21.736716    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx" podUID="a37648fb-1120-4f9d-a25b-2158bbbd3483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 10:03:22.009692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:22.009606    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"
Nov 02 10:03:22.009692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:22.009651    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts" event={"ID":"fea54a6f-c7ec-4a58-8d50-205640bf1e31","Type":"ContainerDied","Data":"f21b63049eb5f7b16e581840e12f51386b969b5096781352ad34bca0ce0f3e53"}
Nov 02 10:03:22.010194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:22.009710    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f21b63049eb5f7b16e581840e12f51386b969b5096781352ad34bca0ce0f3e53"
Nov 02 10:03:22.734714 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:22.734676    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 10:03:22.734898 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:22.734737    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx" podUID="a37648fb-1120-4f9d-a25b-2158bbbd3483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 10:03:23.735008 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:23.734968    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 10:03:23.735526 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:23.735034    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx" podUID="a37648fb-1120-4f9d-a25b-2158bbbd3483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 10:03:24.735172 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:24.735137    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 10:03:24.735706 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:24.735196    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx" podUID="a37648fb-1120-4f9d-a25b-2158bbbd3483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 10:03:25.735449 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:25.735409    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 10:03:25.735870 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:25.735470    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx" podUID="a37648fb-1120-4f9d-a25b-2158bbbd3483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 10:03:25.805731 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:25.805691    4449 scope.go:122] "RemoveContainer" containerID="960191c283185a72e94973b40b89ac092f64e76c9e4f951bbf3249dcbc764bb9"
Nov 02 10:03:26.739594 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:26.739565    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:26.740046 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:26.740023    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:26.740837 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:26.740814    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:03:27.030925 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:27.030827    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"]
Nov 02 10:03:27.037808 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:27.037773    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b828eafc-kt44t"]
Nov 02 10:03:27.206311 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:27.206277    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef68126-35a7-4716-a359-5b31ff36b0cf" path="/var/lib/kubelet/pods/eef68126-35a7-4716-a359-5b31ff36b0cf/volumes"
Nov 02 10:03:28.966404 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:28.966368    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"]
Nov 02 10:03:28.966926 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:28.966912    4449 cpu_manager.go:414] "containerMap: removing container" podUID="fea54a6f-c7ec-4a58-8d50-205640bf1e31" containerName="revoke-keys"
Nov 02 10:03:28.966972 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:28.966927    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea54a6f-c7ec-4a58-8d50-205640bf1e31" containerName="revoke-keys"
Nov 02 10:03:28.967027 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:28.967019    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea54a6f-c7ec-4a58-8d50-205640bf1e31" containerName="revoke-keys"
Nov 02 10:03:28.972209 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:28.972191    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"
Nov 02 10:03:28.978820 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:28.978773    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:03:28.978820 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:28.978773    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 10:03:28.978820 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:28.978782    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:03:28.979061 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:28.978828    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:03:28.979061 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:28.979055    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"]
Nov 02 10:03:29.060658 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:29.060624    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4fsb\" (UniqueName: \"kubernetes.io/projected/a34894f0-d536-4d23-89af-b77ca263154a-kube-api-access-q4fsb\") pod \"maas-api-revoke-sub-f02b017c-6mnhg\" (UID: \"a34894f0-d536-4d23-89af-b77ca263154a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"
Nov 02 10:03:29.060816 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:29.060670    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f02b017c-6mnhg\" (UID: \"a34894f0-d536-4d23-89af-b77ca263154a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"
Nov 02 10:03:29.060816 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:29.060752    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f02b017c-6mnhg\" (UID: \"a34894f0-d536-4d23-89af-b77ca263154a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"
Nov 02 10:03:29.161669 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:29.161626    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4fsb\" (UniqueName: \"kubernetes.io/projected/a34894f0-d536-4d23-89af-b77ca263154a-kube-api-access-q4fsb\") pod \"maas-api-revoke-sub-f02b017c-6mnhg\" (UID: \"a34894f0-d536-4d23-89af-b77ca263154a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"
Nov 02 10:03:29.161669 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:29.161676    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f02b017c-6mnhg\" (UID: \"a34894f0-d536-4d23-89af-b77ca263154a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"
Nov 02 10:03:29.161946 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:29.161708    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f02b017c-6mnhg\" (UID: \"a34894f0-d536-4d23-89af-b77ca263154a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"
Nov 02 10:03:29.162450 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:29.162425    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f02b017c-6mnhg\" (UID: \"a34894f0-d536-4d23-89af-b77ca263154a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"
Nov 02 10:03:29.164286 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:29.164264    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f02b017c-6mnhg\" (UID: \"a34894f0-d536-4d23-89af-b77ca263154a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"
Nov 02 10:03:29.174405 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:29.174386    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4fsb\" (UniqueName: \"kubernetes.io/projected/a34894f0-d536-4d23-89af-b77ca263154a-kube-api-access-q4fsb\") pod \"maas-api-revoke-sub-f02b017c-6mnhg\" (UID: \"a34894f0-d536-4d23-89af-b77ca263154a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"
Nov 02 10:03:29.287374 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:29.287351    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"
Nov 02 10:03:29.568367 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:29.568342    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"]
Nov 02 10:03:29.570714 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:03:29.570688    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34894f0_d536_4d23_89af_b77ca263154a.slice/crio-92dc945b7fde19abfcbbbadaf322ac0c979dbdf11f0a1e3e7e33fbb7bfec7038 WatchSource:0}: Error finding container 92dc945b7fde19abfcbbbadaf322ac0c979dbdf11f0a1e3e7e33fbb7bfec7038: Status 404 returned error can't find the container with id 92dc945b7fde19abfcbbbadaf322ac0c979dbdf11f0a1e3e7e33fbb7bfec7038
Nov 02 10:03:30.062663 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:30.062612    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg" event={"ID":"a34894f0-d536-4d23-89af-b77ca263154a","Type":"ContainerStarted","Data":"9f9135e72823c10dd921c90800636c72deb47253c4f035f792a804146037d7a1"}
Nov 02 10:03:30.063048 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:30.062687    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg" event={"ID":"a34894f0-d536-4d23-89af-b77ca263154a","Type":"ContainerStarted","Data":"92dc945b7fde19abfcbbbadaf322ac0c979dbdf11f0a1e3e7e33fbb7bfec7038"}
Nov 02 10:03:30.082821 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:30.082774    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg" podStartSLOduration=2.08275938 podStartE2EDuration="2.08275938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:03:30.080203282 +0000 UTC m=+1687.018490081" watchObservedRunningTime="2026-11-02 10:03:30.08275938 +0000 UTC m=+1687.021046155"
Nov 02 10:03:31.071362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:31.071325    4449 generic.go:356] "Generic (PLEG): container finished" podID="a34894f0-d536-4d23-89af-b77ca263154a" containerID="9f9135e72823c10dd921c90800636c72deb47253c4f035f792a804146037d7a1" exitCode=0
Nov 02 10:03:31.071932 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:31.071413    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg" event={"ID":"a34894f0-d536-4d23-89af-b77ca263154a","Type":"ContainerDied","Data":"9f9135e72823c10dd921c90800636c72deb47253c4f035f792a804146037d7a1"}
Nov 02 10:03:32.228988 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:32.228966    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"
Nov 02 10:03:32.307459 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:32.307421    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-openshift-ca\") pod \"a34894f0-d536-4d23-89af-b77ca263154a\" (UID: \"a34894f0-d536-4d23-89af-b77ca263154a\") "
Nov 02 10:03:32.307636 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:32.307520    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34894f0-d536-4d23-89af-b77ca263154a-kube-api-access-q4fsb\" (UniqueName: \"kubernetes.io/projected/a34894f0-d536-4d23-89af-b77ca263154a-kube-api-access-q4fsb\") pod \"a34894f0-d536-4d23-89af-b77ca263154a\" (UID: \"a34894f0-d536-4d23-89af-b77ca263154a\") "
Nov 02 10:03:32.307714 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:32.307691    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-xks-ca\") pod \"a34894f0-d536-4d23-89af-b77ca263154a\" (UID: \"a34894f0-d536-4d23-89af-b77ca263154a\") "
Nov 02 10:03:32.307862 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:32.307840    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-openshift-ca" pod "a34894f0-d536-4d23-89af-b77ca263154a" (UID: "a34894f0-d536-4d23-89af-b77ca263154a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:32.308175 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:32.308157    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:32.309817 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:32.309792    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34894f0-d536-4d23-89af-b77ca263154a-kube-api-access-q4fsb" pod "a34894f0-d536-4d23-89af-b77ca263154a" (UID: "a34894f0-d536-4d23-89af-b77ca263154a"). InnerVolumeSpecName "kube-api-access-q4fsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:32.309963 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:32.309945    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-xks-ca" pod "a34894f0-d536-4d23-89af-b77ca263154a" (UID: "a34894f0-d536-4d23-89af-b77ca263154a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:03:32.409941 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:32.409847    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4fsb\" (UniqueName: \"kubernetes.io/projected/a34894f0-d536-4d23-89af-b77ca263154a-kube-api-access-q4fsb\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:32.409941 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:32.409887    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34894f0-d536-4d23-89af-b77ca263154a-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:33.086354 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:33.086321    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"
Nov 02 10:03:33.086617 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:33.086353    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg" event={"ID":"a34894f0-d536-4d23-89af-b77ca263154a","Type":"ContainerDied","Data":"92dc945b7fde19abfcbbbadaf322ac0c979dbdf11f0a1e3e7e33fbb7bfec7038"}
Nov 02 10:03:33.086617 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:33.086425    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92dc945b7fde19abfcbbbadaf322ac0c979dbdf11f0a1e3e7e33fbb7bfec7038"
Nov 02 10:03:36.549119 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:03:36.549082    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:03:38.032254 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:38.032191    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"]
Nov 02 10:03:38.035591 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:38.035564    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d429e290-m8jz7"]
Nov 02 10:03:39.206712 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:39.206674    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776a2c05-d364-450b-87ed-fbcf7554a5c2" path="/var/lib/kubelet/pods/776a2c05-d364-450b-87ed-fbcf7554a5c2/volumes"
Nov 02 10:03:43.343267 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.343210    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"]
Nov 02 10:03:43.343777 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.343762    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a34894f0-d536-4d23-89af-b77ca263154a" containerName="revoke-keys"
Nov 02 10:03:43.343823 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.343779    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a34894f0-d536-4d23-89af-b77ca263154a" containerName="revoke-keys"
Nov 02 10:03:43.343882 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.343872    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a34894f0-d536-4d23-89af-b77ca263154a" containerName="revoke-keys"
Nov 02 10:03:43.355926 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.355889    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"
Nov 02 10:03:43.360091 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.360057    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:03:43.360581 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.360549    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"]
Nov 02 10:03:43.361090 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.361066    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:03:43.361291 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.361114    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:03:43.361385 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.361133    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 10:03:43.471357 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.471313    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7de8b560-m6q54\" (UID: \"f99c0175-8978-42bd-aebe-e59afa74d216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"
Nov 02 10:03:43.471550 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.471436    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hc74\" (UniqueName: \"kubernetes.io/projected/f99c0175-8978-42bd-aebe-e59afa74d216-kube-api-access-7hc74\") pod \"maas-api-revoke-sub-7de8b560-m6q54\" (UID: \"f99c0175-8978-42bd-aebe-e59afa74d216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"
Nov 02 10:03:43.471550 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.471472    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7de8b560-m6q54\" (UID: \"f99c0175-8978-42bd-aebe-e59afa74d216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"
Nov 02 10:03:43.572834 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.572792    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7de8b560-m6q54\" (UID: \"f99c0175-8978-42bd-aebe-e59afa74d216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"
Nov 02 10:03:43.573034 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.572910    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hc74\" (UniqueName: \"kubernetes.io/projected/f99c0175-8978-42bd-aebe-e59afa74d216-kube-api-access-7hc74\") pod \"maas-api-revoke-sub-7de8b560-m6q54\" (UID: \"f99c0175-8978-42bd-aebe-e59afa74d216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"
Nov 02 10:03:43.573034 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.572959    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7de8b560-m6q54\" (UID: \"f99c0175-8978-42bd-aebe-e59afa74d216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"
Nov 02 10:03:43.573565 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.573534    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7de8b560-m6q54\" (UID: \"f99c0175-8978-42bd-aebe-e59afa74d216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"
Nov 02 10:03:43.575495 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.575471    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7de8b560-m6q54\" (UID: \"f99c0175-8978-42bd-aebe-e59afa74d216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"
Nov 02 10:03:43.585313 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.585283    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hc74\" (UniqueName: \"kubernetes.io/projected/f99c0175-8978-42bd-aebe-e59afa74d216-kube-api-access-7hc74\") pod \"maas-api-revoke-sub-7de8b560-m6q54\" (UID: \"f99c0175-8978-42bd-aebe-e59afa74d216\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"
Nov 02 10:03:43.676309 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.676174    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"
Nov 02 10:03:43.958566 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:43.958536    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"]
Nov 02 10:03:43.959269 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:03:43.959243    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99c0175_8978_42bd_aebe_e59afa74d216.slice/crio-cbf45291655028fb69f414bc25b754142a06148fb03659f1167fee368b691e3d WatchSource:0}: Error finding container cbf45291655028fb69f414bc25b754142a06148fb03659f1167fee368b691e3d: Status 404 returned error can't find the container with id cbf45291655028fb69f414bc25b754142a06148fb03659f1167fee368b691e3d
Nov 02 10:03:44.173275 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:44.173205    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54" event={"ID":"f99c0175-8978-42bd-aebe-e59afa74d216","Type":"ContainerStarted","Data":"cbf45291655028fb69f414bc25b754142a06148fb03659f1167fee368b691e3d"}
Nov 02 10:03:45.232894 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:45.232278    4449 generic.go:356] "Generic (PLEG): container finished" podID="f99c0175-8978-42bd-aebe-e59afa74d216" containerID="e28ea232e59a365eb71115096d7c317dd3474dfd9e4c0cc28d64e84a47e90dc6" exitCode=0
Nov 02 10:03:45.248788 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:45.248741    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54" event={"ID":"f99c0175-8978-42bd-aebe-e59afa74d216","Type":"ContainerDied","Data":"e28ea232e59a365eb71115096d7c317dd3474dfd9e4c0cc28d64e84a47e90dc6"}
Nov 02 10:03:46.383404 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:46.383375    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"
Nov 02 10:03:46.525419 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:46.525330    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99c0175-8978-42bd-aebe-e59afa74d216-kube-api-access-7hc74\" (UniqueName: \"kubernetes.io/projected/f99c0175-8978-42bd-aebe-e59afa74d216-kube-api-access-7hc74\") pod \"f99c0175-8978-42bd-aebe-e59afa74d216\" (UID: \"f99c0175-8978-42bd-aebe-e59afa74d216\") "
Nov 02 10:03:46.525585 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:46.525489    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-xks-ca\") pod \"f99c0175-8978-42bd-aebe-e59afa74d216\" (UID: \"f99c0175-8978-42bd-aebe-e59afa74d216\") "
Nov 02 10:03:46.525585 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:46.525521    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-openshift-ca\") pod \"f99c0175-8978-42bd-aebe-e59afa74d216\" (UID: \"f99c0175-8978-42bd-aebe-e59afa74d216\") "
Nov 02 10:03:46.525898 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:46.525865    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-openshift-ca" pod "f99c0175-8978-42bd-aebe-e59afa74d216" (UID: "f99c0175-8978-42bd-aebe-e59afa74d216"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:46.527758 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:46.527734    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-xks-ca" pod "f99c0175-8978-42bd-aebe-e59afa74d216" (UID: "f99c0175-8978-42bd-aebe-e59afa74d216"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:03:46.527876 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:46.527801    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99c0175-8978-42bd-aebe-e59afa74d216-kube-api-access-7hc74" pod "f99c0175-8978-42bd-aebe-e59afa74d216" (UID: "f99c0175-8978-42bd-aebe-e59afa74d216"). InnerVolumeSpecName "kube-api-access-7hc74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:46.627108 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:46.627065    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:46.627108 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:46.627102    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99c0175-8978-42bd-aebe-e59afa74d216-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:46.627108 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:46.627116    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hc74\" (UniqueName: \"kubernetes.io/projected/f99c0175-8978-42bd-aebe-e59afa74d216-kube-api-access-7hc74\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:47.247445 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:47.247408    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54" event={"ID":"f99c0175-8978-42bd-aebe-e59afa74d216","Type":"ContainerDied","Data":"cbf45291655028fb69f414bc25b754142a06148fb03659f1167fee368b691e3d"}
Nov 02 10:03:47.247616 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:47.247458    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbf45291655028fb69f414bc25b754142a06148fb03659f1167fee368b691e3d"
Nov 02 10:03:47.247616 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:47.247422    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"
Nov 02 10:03:48.026951 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:48.026915    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"]
Nov 02 10:03:48.029988 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:48.029957    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19a2d3cc-5dh7h"]
Nov 02 10:03:49.207137 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:49.207099    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f73a74-81b9-4231-938d-404a42335a88" path="/var/lib/kubelet/pods/d8f73a74-81b9-4231-938d-404a42335a88/volumes"
Nov 02 10:03:53.893491 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:53.893447    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"]
Nov 02 10:03:53.894001 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:53.893987    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f99c0175-8978-42bd-aebe-e59afa74d216" containerName="revoke-keys"
Nov 02 10:03:53.894052 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:53.894002    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99c0175-8978-42bd-aebe-e59afa74d216" containerName="revoke-keys"
Nov 02 10:03:53.894109 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:53.894100    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99c0175-8978-42bd-aebe-e59afa74d216" containerName="revoke-keys"
Nov 02 10:03:53.898624 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:53.898606    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"
Nov 02 10:03:53.903630 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:53.903600    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:03:53.903758 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:53.903719    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 10:03:53.903831 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:53.903800    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:03:53.903938 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:53.903907    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:03:53.908013 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:53.907983    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"]
Nov 02 10:03:54.012749 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:54.012715    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2w4c\" (UniqueName: \"kubernetes.io/projected/0991bf33-2095-4216-bbe2-6444e1de285f-kube-api-access-b2w4c\") pod \"maas-api-revoke-sub-d1735450-kljcj\" (UID: \"0991bf33-2095-4216-bbe2-6444e1de285f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"
Nov 02 10:03:54.012941 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:54.012776    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1735450-kljcj\" (UID: \"0991bf33-2095-4216-bbe2-6444e1de285f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"
Nov 02 10:03:54.012992 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:54.012931    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1735450-kljcj\" (UID: \"0991bf33-2095-4216-bbe2-6444e1de285f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"
Nov 02 10:03:54.113915 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:54.113884    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2w4c\" (UniqueName: \"kubernetes.io/projected/0991bf33-2095-4216-bbe2-6444e1de285f-kube-api-access-b2w4c\") pod \"maas-api-revoke-sub-d1735450-kljcj\" (UID: \"0991bf33-2095-4216-bbe2-6444e1de285f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"
Nov 02 10:03:54.114110 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:54.113941    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1735450-kljcj\" (UID: \"0991bf33-2095-4216-bbe2-6444e1de285f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"
Nov 02 10:03:54.114110 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:54.114001    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1735450-kljcj\" (UID: \"0991bf33-2095-4216-bbe2-6444e1de285f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"
Nov 02 10:03:54.114783 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:54.114760    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1735450-kljcj\" (UID: \"0991bf33-2095-4216-bbe2-6444e1de285f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"
Nov 02 10:03:54.116554 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:54.116533    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1735450-kljcj\" (UID: \"0991bf33-2095-4216-bbe2-6444e1de285f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"
Nov 02 10:03:54.127586 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:54.127561    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2w4c\" (UniqueName: \"kubernetes.io/projected/0991bf33-2095-4216-bbe2-6444e1de285f-kube-api-access-b2w4c\") pod \"maas-api-revoke-sub-d1735450-kljcj\" (UID: \"0991bf33-2095-4216-bbe2-6444e1de285f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"
Nov 02 10:03:54.214772 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:54.214689    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"
Nov 02 10:03:54.499101 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:54.499073    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"]
Nov 02 10:03:54.500319 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:03:54.500290    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0991bf33_2095_4216_bbe2_6444e1de285f.slice/crio-d6fc98320161cde62d3f111ed5f4a0a21172289a2e354b3d3590124b8f616aa4 WatchSource:0}: Error finding container d6fc98320161cde62d3f111ed5f4a0a21172289a2e354b3d3590124b8f616aa4: Status 404 returned error can't find the container with id d6fc98320161cde62d3f111ed5f4a0a21172289a2e354b3d3590124b8f616aa4
Nov 02 10:03:55.381066 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:55.380839    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj" event={"ID":"0991bf33-2095-4216-bbe2-6444e1de285f","Type":"ContainerStarted","Data":"cd53752929d7ff906b9076e56a09b723786b764723d56bfdf1c739a94d2ea178"}
Nov 02 10:03:55.382730 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:55.382690    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj" event={"ID":"0991bf33-2095-4216-bbe2-6444e1de285f","Type":"ContainerStarted","Data":"d6fc98320161cde62d3f111ed5f4a0a21172289a2e354b3d3590124b8f616aa4"}
Nov 02 10:03:55.418246 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:55.415108    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj" podStartSLOduration=2.415084387 podStartE2EDuration="2.415084387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:03:55.409685538 +0000 UTC m=+1712.347972315" watchObservedRunningTime="2026-11-02 10:03:55.415084387 +0000 UTC m=+1712.353371165"
Nov 02 10:03:56.389412 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:56.389378    4449 generic.go:356] "Generic (PLEG): container finished" podID="0991bf33-2095-4216-bbe2-6444e1de285f" containerID="cd53752929d7ff906b9076e56a09b723786b764723d56bfdf1c739a94d2ea178" exitCode=0
Nov 02 10:03:56.390024 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:56.389661    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj" event={"ID":"0991bf33-2095-4216-bbe2-6444e1de285f","Type":"ContainerDied","Data":"cd53752929d7ff906b9076e56a09b723786b764723d56bfdf1c739a94d2ea178"}
Nov 02 10:03:57.550121 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:57.550092    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"
Nov 02 10:03:57.584553 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:57.584519    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-openshift-ca\") pod \"0991bf33-2095-4216-bbe2-6444e1de285f\" (UID: \"0991bf33-2095-4216-bbe2-6444e1de285f\") "
Nov 02 10:03:57.584741 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:57.584670    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0991bf33-2095-4216-bbe2-6444e1de285f-kube-api-access-b2w4c\" (UniqueName: \"kubernetes.io/projected/0991bf33-2095-4216-bbe2-6444e1de285f-kube-api-access-b2w4c\") pod \"0991bf33-2095-4216-bbe2-6444e1de285f\" (UID: \"0991bf33-2095-4216-bbe2-6444e1de285f\") "
Nov 02 10:03:57.584807 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:57.584766    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-xks-ca\") pod \"0991bf33-2095-4216-bbe2-6444e1de285f\" (UID: \"0991bf33-2095-4216-bbe2-6444e1de285f\") "
Nov 02 10:03:57.584973 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:57.584942    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-openshift-ca" pod "0991bf33-2095-4216-bbe2-6444e1de285f" (UID: "0991bf33-2095-4216-bbe2-6444e1de285f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:57.585298 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:57.585276    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:57.586987 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:57.586961    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-xks-ca" pod "0991bf33-2095-4216-bbe2-6444e1de285f" (UID: "0991bf33-2095-4216-bbe2-6444e1de285f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:03:57.587094 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:57.587042    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0991bf33-2095-4216-bbe2-6444e1de285f-kube-api-access-b2w4c" pod "0991bf33-2095-4216-bbe2-6444e1de285f" (UID: "0991bf33-2095-4216-bbe2-6444e1de285f"). InnerVolumeSpecName "kube-api-access-b2w4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:57.686499 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:57.686414    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0991bf33-2095-4216-bbe2-6444e1de285f-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:57.686499 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:57.686445    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2w4c\" (UniqueName: \"kubernetes.io/projected/0991bf33-2095-4216-bbe2-6444e1de285f-kube-api-access-b2w4c\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:58.404881 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:58.404840    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj" event={"ID":"0991bf33-2095-4216-bbe2-6444e1de285f","Type":"ContainerDied","Data":"d6fc98320161cde62d3f111ed5f4a0a21172289a2e354b3d3590124b8f616aa4"}
Nov 02 10:03:58.405071 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:58.404893    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6fc98320161cde62d3f111ed5f4a0a21172289a2e354b3d3590124b8f616aa4"
Nov 02 10:03:58.405071 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:03:58.404857    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"
Nov 02 10:04:01.024121 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:01.024076    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"]
Nov 02 10:04:01.027232 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:01.027193    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40f185ba-7zbgd"]
Nov 02 10:04:01.205716 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:01.205680    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7701534c-2a3d-4211-a06a-2af4a6b454f7" path="/var/lib/kubelet/pods/7701534c-2a3d-4211-a06a-2af4a6b454f7/volumes"
Nov 02 10:04:06.000012 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:05.999966    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"]
Nov 02 10:04:06.000686 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.000508    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0991bf33-2095-4216-bbe2-6444e1de285f" containerName="revoke-keys"
Nov 02 10:04:06.000686 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.000521    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0991bf33-2095-4216-bbe2-6444e1de285f" containerName="revoke-keys"
Nov 02 10:04:06.000686 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.000615    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0991bf33-2095-4216-bbe2-6444e1de285f" containerName="revoke-keys"
Nov 02 10:04:06.005377 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.005355    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"
Nov 02 10:04:06.011987 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.011957    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:04:06.012153 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.012048    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 10:04:06.012384 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.012286    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:04:06.012384 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.011430    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:04:06.013693 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.013666    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"]
Nov 02 10:04:06.104425 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.104386    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfgh\" (UniqueName: \"kubernetes.io/projected/8d018347-76ec-437b-9bb4-3dd973a79c71-kube-api-access-xmfgh\") pod \"maas-api-revoke-sub-558229dc-qjwnj\" (UID: \"8d018347-76ec-437b-9bb4-3dd973a79c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"
Nov 02 10:04:06.104650 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.104533    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-558229dc-qjwnj\" (UID: \"8d018347-76ec-437b-9bb4-3dd973a79c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"
Nov 02 10:04:06.104734 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.104653    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-558229dc-qjwnj\" (UID: \"8d018347-76ec-437b-9bb4-3dd973a79c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"
Nov 02 10:04:06.205608 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.205565    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfgh\" (UniqueName: \"kubernetes.io/projected/8d018347-76ec-437b-9bb4-3dd973a79c71-kube-api-access-xmfgh\") pod \"maas-api-revoke-sub-558229dc-qjwnj\" (UID: \"8d018347-76ec-437b-9bb4-3dd973a79c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"
Nov 02 10:04:06.205799 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.205678    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-558229dc-qjwnj\" (UID: \"8d018347-76ec-437b-9bb4-3dd973a79c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"
Nov 02 10:04:06.205799 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.205740    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-558229dc-qjwnj\" (UID: \"8d018347-76ec-437b-9bb4-3dd973a79c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"
Nov 02 10:04:06.206412 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.206391    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-558229dc-qjwnj\" (UID: \"8d018347-76ec-437b-9bb4-3dd973a79c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"
Nov 02 10:04:06.208356 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.208331    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-558229dc-qjwnj\" (UID: \"8d018347-76ec-437b-9bb4-3dd973a79c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"
Nov 02 10:04:06.220735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.220698    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfgh\" (UniqueName: \"kubernetes.io/projected/8d018347-76ec-437b-9bb4-3dd973a79c71-kube-api-access-xmfgh\") pod \"maas-api-revoke-sub-558229dc-qjwnj\" (UID: \"8d018347-76ec-437b-9bb4-3dd973a79c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"
Nov 02 10:04:06.326777 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.326744    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"
Nov 02 10:04:06.547340 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:04:06.547295    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:04:06.667391 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:06.667344    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"]
Nov 02 10:04:06.668069 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:04:06.668039    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d018347_76ec_437b_9bb4_3dd973a79c71.slice/crio-d45308a9c6a8ffafcff36e27668303710f692412570cad947cc245656f80ad52 WatchSource:0}: Error finding container d45308a9c6a8ffafcff36e27668303710f692412570cad947cc245656f80ad52: Status 404 returned error can't find the container with id d45308a9c6a8ffafcff36e27668303710f692412570cad947cc245656f80ad52
Nov 02 10:04:07.558436 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:07.558395    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj" event={"ID":"8d018347-76ec-437b-9bb4-3dd973a79c71","Type":"ContainerStarted","Data":"8dc95bb0ab2ffcafbaf528b62600174e8b17b5ad213d87e1188afe63e36b90bc"}
Nov 02 10:04:07.558915 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:07.558495    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj" event={"ID":"8d018347-76ec-437b-9bb4-3dd973a79c71","Type":"ContainerStarted","Data":"d45308a9c6a8ffafcff36e27668303710f692412570cad947cc245656f80ad52"}
Nov 02 10:04:07.580791 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:07.580725    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj" podStartSLOduration=2.580708186 podStartE2EDuration="2.580708186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:04:07.576141675 +0000 UTC m=+1724.514428453" watchObservedRunningTime="2026-11-02 10:04:07.580708186 +0000 UTC m=+1724.518994963"
Nov 02 10:04:08.566708 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:08.566671    4449 generic.go:356] "Generic (PLEG): container finished" podID="8d018347-76ec-437b-9bb4-3dd973a79c71" containerID="8dc95bb0ab2ffcafbaf528b62600174e8b17b5ad213d87e1188afe63e36b90bc" exitCode=0
Nov 02 10:04:08.567133 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:08.566753    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj" event={"ID":"8d018347-76ec-437b-9bb4-3dd973a79c71","Type":"ContainerDied","Data":"8dc95bb0ab2ffcafbaf528b62600174e8b17b5ad213d87e1188afe63e36b90bc"}
Nov 02 10:04:09.713875 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:09.713838    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"
Nov 02 10:04:09.853552 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:09.853462    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d018347-76ec-437b-9bb4-3dd973a79c71-kube-api-access-xmfgh\" (UniqueName: \"kubernetes.io/projected/8d018347-76ec-437b-9bb4-3dd973a79c71-kube-api-access-xmfgh\") pod \"8d018347-76ec-437b-9bb4-3dd973a79c71\" (UID: \"8d018347-76ec-437b-9bb4-3dd973a79c71\") "
Nov 02 10:04:09.853711 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:09.853619    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-xks-ca\") pod \"8d018347-76ec-437b-9bb4-3dd973a79c71\" (UID: \"8d018347-76ec-437b-9bb4-3dd973a79c71\") "
Nov 02 10:04:09.853711 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:09.853652    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-openshift-ca\") pod \"8d018347-76ec-437b-9bb4-3dd973a79c71\" (UID: \"8d018347-76ec-437b-9bb4-3dd973a79c71\") "
Nov 02 10:04:09.854083 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:09.854053    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-openshift-ca" pod "8d018347-76ec-437b-9bb4-3dd973a79c71" (UID: "8d018347-76ec-437b-9bb4-3dd973a79c71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:04:09.855847 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:09.855821    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-xks-ca" pod "8d018347-76ec-437b-9bb4-3dd973a79c71" (UID: "8d018347-76ec-437b-9bb4-3dd973a79c71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:04:09.855941 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:09.855841    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d018347-76ec-437b-9bb4-3dd973a79c71-kube-api-access-xmfgh" pod "8d018347-76ec-437b-9bb4-3dd973a79c71" (UID: "8d018347-76ec-437b-9bb4-3dd973a79c71"). InnerVolumeSpecName "kube-api-access-xmfgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:04:09.955460 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:09.955420    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:09.955460 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:09.955455    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d018347-76ec-437b-9bb4-3dd973a79c71-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:09.955460 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:09.955466    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmfgh\" (UniqueName: \"kubernetes.io/projected/8d018347-76ec-437b-9bb4-3dd973a79c71-kube-api-access-xmfgh\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:10.580714 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:10.580676    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj" event={"ID":"8d018347-76ec-437b-9bb4-3dd973a79c71","Type":"ContainerDied","Data":"d45308a9c6a8ffafcff36e27668303710f692412570cad947cc245656f80ad52"}
Nov 02 10:04:10.580924 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:10.580734    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d45308a9c6a8ffafcff36e27668303710f692412570cad947cc245656f80ad52"
Nov 02 10:04:10.580924 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:10.580685    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"
Nov 02 10:04:14.042572 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:14.042538    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"]
Nov 02 10:04:14.048698 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:14.048667    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cc5919-fhb2t"]
Nov 02 10:04:15.215065 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:15.215025    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e107f26f-04b1-44e5-ad54-1e42fc4c09ea" path="/var/lib/kubelet/pods/e107f26f-04b1-44e5-ad54-1e42fc4c09ea/volumes"
Nov 02 10:04:16.566876 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.566835    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"]
Nov 02 10:04:16.567581 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.567559    4449 cpu_manager.go:414] "containerMap: removing container" podUID="8d018347-76ec-437b-9bb4-3dd973a79c71" containerName="revoke-keys"
Nov 02 10:04:16.567581 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.567580    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d018347-76ec-437b-9bb4-3dd973a79c71" containerName="revoke-keys"
Nov 02 10:04:16.567760 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.567660    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d018347-76ec-437b-9bb4-3dd973a79c71" containerName="revoke-keys"
Nov 02 10:04:16.571070 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.571047    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"
Nov 02 10:04:16.576587 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.576564    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:04:16.576760 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.576605    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:04:16.580621 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.580598    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"]
Nov 02 10:04:16.622545 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.622514    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:04:16.622739 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.622590    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 10:04:16.733361 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.733318    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz6kn\" (UniqueName: \"kubernetes.io/projected/1cc0e764-96e1-46cf-a580-96253011d988-kube-api-access-hz6kn\") pod \"maas-api-revoke-sub-4fc0c347-28688\" (UID: \"1cc0e764-96e1-46cf-a580-96253011d988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"
Nov 02 10:04:16.733828 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.733374    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fc0c347-28688\" (UID: \"1cc0e764-96e1-46cf-a580-96253011d988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"
Nov 02 10:04:16.733828 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.733588    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fc0c347-28688\" (UID: \"1cc0e764-96e1-46cf-a580-96253011d988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"
Nov 02 10:04:16.834948 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.834848    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz6kn\" (UniqueName: \"kubernetes.io/projected/1cc0e764-96e1-46cf-a580-96253011d988-kube-api-access-hz6kn\") pod \"maas-api-revoke-sub-4fc0c347-28688\" (UID: \"1cc0e764-96e1-46cf-a580-96253011d988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"
Nov 02 10:04:16.834948 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.834899    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fc0c347-28688\" (UID: \"1cc0e764-96e1-46cf-a580-96253011d988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"
Nov 02 10:04:16.835197 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.835044    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fc0c347-28688\" (UID: \"1cc0e764-96e1-46cf-a580-96253011d988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"
Nov 02 10:04:16.835646 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.835610    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fc0c347-28688\" (UID: \"1cc0e764-96e1-46cf-a580-96253011d988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"
Nov 02 10:04:16.837657 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.837624    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fc0c347-28688\" (UID: \"1cc0e764-96e1-46cf-a580-96253011d988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"
Nov 02 10:04:16.847026 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.847002    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz6kn\" (UniqueName: \"kubernetes.io/projected/1cc0e764-96e1-46cf-a580-96253011d988-kube-api-access-hz6kn\") pod \"maas-api-revoke-sub-4fc0c347-28688\" (UID: \"1cc0e764-96e1-46cf-a580-96253011d988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"
Nov 02 10:04:16.893298 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:16.893260    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"
Nov 02 10:04:17.172051 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:17.172018    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"]
Nov 02 10:04:17.172875 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:04:17.172842    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc0e764_96e1_46cf_a580_96253011d988.slice/crio-3f7eec8c3deef4a0651f81e7395daafae1f8c34226094c7a96560275f7430467 WatchSource:0}: Error finding container 3f7eec8c3deef4a0651f81e7395daafae1f8c34226094c7a96560275f7430467: Status 404 returned error can't find the container with id 3f7eec8c3deef4a0651f81e7395daafae1f8c34226094c7a96560275f7430467
Nov 02 10:04:17.637912 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:17.637867    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688" event={"ID":"1cc0e764-96e1-46cf-a580-96253011d988","Type":"ContainerStarted","Data":"51c107d38e56ad7f29e8c76746e9a7c9878d2e4273164e8878ebc6a719b7c4cf"}
Nov 02 10:04:17.638472 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:17.637920    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688" event={"ID":"1cc0e764-96e1-46cf-a580-96253011d988","Type":"ContainerStarted","Data":"3f7eec8c3deef4a0651f81e7395daafae1f8c34226094c7a96560275f7430467"}
Nov 02 10:04:17.658714 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:17.658657    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688" podStartSLOduration=1.658642983 podStartE2EDuration="1.658642983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:04:17.655995532 +0000 UTC m=+1734.594282319" watchObservedRunningTime="2026-11-02 10:04:17.658642983 +0000 UTC m=+1734.596929819"
Nov 02 10:04:18.653157 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:18.653117    4449 generic.go:356] "Generic (PLEG): container finished" podID="1cc0e764-96e1-46cf-a580-96253011d988" containerID="51c107d38e56ad7f29e8c76746e9a7c9878d2e4273164e8878ebc6a719b7c4cf" exitCode=0
Nov 02 10:04:18.653904 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:18.653868    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688" event={"ID":"1cc0e764-96e1-46cf-a580-96253011d988","Type":"ContainerDied","Data":"51c107d38e56ad7f29e8c76746e9a7c9878d2e4273164e8878ebc6a719b7c4cf"}
Nov 02 10:04:19.919575 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:19.919297    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"
Nov 02 10:04:19.995676 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:19.995627    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc0e764-96e1-46cf-a580-96253011d988-kube-api-access-hz6kn\" (UniqueName: \"kubernetes.io/projected/1cc0e764-96e1-46cf-a580-96253011d988-kube-api-access-hz6kn\") pod \"1cc0e764-96e1-46cf-a580-96253011d988\" (UID: \"1cc0e764-96e1-46cf-a580-96253011d988\") "
Nov 02 10:04:19.995887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:19.995724    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-openshift-ca\") pod \"1cc0e764-96e1-46cf-a580-96253011d988\" (UID: \"1cc0e764-96e1-46cf-a580-96253011d988\") "
Nov 02 10:04:19.995962 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:19.995887    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-xks-ca\") pod \"1cc0e764-96e1-46cf-a580-96253011d988\" (UID: \"1cc0e764-96e1-46cf-a580-96253011d988\") "
Nov 02 10:04:19.996094 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:19.996071    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-openshift-ca" pod "1cc0e764-96e1-46cf-a580-96253011d988" (UID: "1cc0e764-96e1-46cf-a580-96253011d988"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:04:19.996419 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:19.996399    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:19.998114 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:19.998082    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc0e764-96e1-46cf-a580-96253011d988-kube-api-access-hz6kn" pod "1cc0e764-96e1-46cf-a580-96253011d988" (UID: "1cc0e764-96e1-46cf-a580-96253011d988"). InnerVolumeSpecName "kube-api-access-hz6kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:04:19.998114 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:19.998090    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-xks-ca" pod "1cc0e764-96e1-46cf-a580-96253011d988" (UID: "1cc0e764-96e1-46cf-a580-96253011d988"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:04:20.097138 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:20.097091    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc0e764-96e1-46cf-a580-96253011d988-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:20.097138 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:20.097133    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz6kn\" (UniqueName: \"kubernetes.io/projected/1cc0e764-96e1-46cf-a580-96253011d988-kube-api-access-hz6kn\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:20.668101 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:20.668063    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"
Nov 02 10:04:20.668318 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:20.668106    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688" event={"ID":"1cc0e764-96e1-46cf-a580-96253011d988","Type":"ContainerDied","Data":"3f7eec8c3deef4a0651f81e7395daafae1f8c34226094c7a96560275f7430467"}
Nov 02 10:04:20.668318 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:20.668156    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f7eec8c3deef4a0651f81e7395daafae1f8c34226094c7a96560275f7430467"
Nov 02 10:04:25.832914 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:25.832886    4449 scope.go:122] "RemoveContainer" containerID="006c97ce6e255ad126ea9d51bd680ca42f4f44e8eaed274a65ef5eeeef7b019e"
Nov 02 10:04:25.843689 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:25.843668    4449 scope.go:122] "RemoveContainer" containerID="97255e58a65324de0772bc0ef27ea4fdfc76cce5f36eab51ac96d38ce8af2bc9"
Nov 02 10:04:25.853037 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:25.853017    4449 scope.go:122] "RemoveContainer" containerID="2581debc8b07cafe64a9d234d8d3e9f47df42c554d32715a23200d02d52bab98"
Nov 02 10:04:25.862085 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:25.862067    4449 scope.go:122] "RemoveContainer" containerID="cb46dec60164a6575c5ce207626b07921e88eb8091e9f9231a21c5908aec0d83"
Nov 02 10:04:25.870601 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:25.870583    4449 scope.go:122] "RemoveContainer" containerID="849d183ac2b232a7c3bebe2b33e67c1e8141f050ae7ef38d7ed6620123f2d23e"
Nov 02 10:04:36.547991 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:04:36.547911    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:04:42.735265 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:42.735204    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"]
Nov 02 10:04:42.750135 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:42.750096    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"]
Nov 02 10:04:47.735539 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:47.735484    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv" podUID="d679fb2c-7057-4104-9b55-3cfa4fcde09c" containerName="payload-processing" containerID="cri-o://1040990dd13ddc3f638a1ba13e02434d80f61c72d61f006e7745505cf84469d6" gracePeriod=25
Nov 02 10:04:47.751294 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:47.751207    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79" podUID="db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f" containerName="payload-pre-processing" containerID="cri-o://308bb8e5c0caa410f11cf281122f6d5722f82285c5ccaacb3fdb877c7d707fb5" gracePeriod=25
Nov 02 10:04:49.587562 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:49.587520    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79" podUID="db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:04:49.587935 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:49.587566    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv" podUID="d679fb2c-7057-4104-9b55-3cfa4fcde09c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:04:50.910355 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:50.910307    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"]
Nov 02 10:04:50.910838 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:50.910592    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j" podUID="f93519ae-58de-4fe9-bbdf-fd2d607d497c" containerName="istio-proxy" containerID="cri-o://47e171145fb459186ba8bd726848b362cf44d6c4ba37888fdd4d75677015d215" gracePeriod=30
Nov 02 10:04:50.912253 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:50.912231    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"]
Nov 02 10:04:50.912448 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:50.912421    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s" podUID="4b6a26be-45d7-47e9-a62e-5920e516f037" containerName="istio-proxy" containerID="cri-o://aa8855b994a1ed196f1597e5ff7285733bf215c8a77e395ac384ba76fccdaa9c" gracePeriod=30
Nov 02 10:04:55.723996 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:55.723953    4449 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 10:04:55.724512 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:55.724018    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j" podUID="f93519ae-58de-4fe9-bbdf-fd2d607d497c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 10:04:56.176578 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.176553    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 10:04:56.217834 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.217806    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 10:04:56.339241 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339173    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-podinfo\") pod \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") "
Nov 02 10:04:56.339458 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339285    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-socket\") pod \"4b6a26be-45d7-47e9-a62e-5920e516f037\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") "
Nov 02 10:04:56.339458 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339324    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-certs\") pod \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") "
Nov 02 10:04:56.339458 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339351    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-token\" (UniqueName: \"kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-token\") pod \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") "
Nov 02 10:04:56.339458 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339413    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-envoy\") pod \"4b6a26be-45d7-47e9-a62e-5920e516f037\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") "
Nov 02 10:04:56.339458 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339441    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-kube-api-access-95d8r\" (UniqueName: \"kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-kube-api-access-95d8r\") pod \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") "
Nov 02 10:04:56.339757 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339479    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-socket\") pod \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") "
Nov 02 10:04:56.339757 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339522    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-credential-socket\") pod \"4b6a26be-45d7-47e9-a62e-5920e516f037\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") "
Nov 02 10:04:56.339757 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339553    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-data\") pod \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") "
Nov 02 10:04:56.339757 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339592    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istiod-ca-cert\") pod \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") "
Nov 02 10:04:56.339757 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339607    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-socket" pod "4b6a26be-45d7-47e9-a62e-5920e516f037" (UID: "4b6a26be-45d7-47e9-a62e-5920e516f037"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.339757 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339627    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-certs" pod "f93519ae-58de-4fe9-bbdf-fd2d607d497c" (UID: "f93519ae-58de-4fe9-bbdf-fd2d607d497c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.339757 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339663    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b6a26be-45d7-47e9-a62e-5920e516f037-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6a26be-45d7-47e9-a62e-5920e516f037-istiod-ca-cert\") pod \"4b6a26be-45d7-47e9-a62e-5920e516f037\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") "
Nov 02 10:04:56.339757 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339698    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-certs\") pod \"4b6a26be-45d7-47e9-a62e-5920e516f037\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") "
Nov 02 10:04:56.340176 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339770    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-podinfo\") pod \"4b6a26be-45d7-47e9-a62e-5920e516f037\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") "
Nov 02 10:04:56.340176 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339813    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-credential-socket\") pod \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") "
Nov 02 10:04:56.340176 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339846    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-envoy\") pod \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\" (UID: \"f93519ae-58de-4fe9-bbdf-fd2d607d497c\") "
Nov 02 10:04:56.340176 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339882    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-kube-api-access-s6t2h\" (UniqueName: \"kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-kube-api-access-s6t2h\") pod \"4b6a26be-45d7-47e9-a62e-5920e516f037\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") "
Nov 02 10:04:56.340176 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339947    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-token\" (UniqueName: \"kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-token\") pod \"4b6a26be-45d7-47e9-a62e-5920e516f037\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") "
Nov 02 10:04:56.340176 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339962    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-data" pod "f93519ae-58de-4fe9-bbdf-fd2d607d497c" (UID: "f93519ae-58de-4fe9-bbdf-fd2d607d497c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.340176 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339975    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-data\") pod \"4b6a26be-45d7-47e9-a62e-5920e516f037\" (UID: \"4b6a26be-45d7-47e9-a62e-5920e516f037\") "
Nov 02 10:04:56.340176 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.339928    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-credential-socket" pod "4b6a26be-45d7-47e9-a62e-5920e516f037" (UID: "4b6a26be-45d7-47e9-a62e-5920e516f037"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.340176 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340152    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istiod-ca-cert" pod "f93519ae-58de-4fe9-bbdf-fd2d607d497c" (UID: "f93519ae-58de-4fe9-bbdf-fd2d607d497c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:04:56.340713 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340360    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-socket" pod "f93519ae-58de-4fe9-bbdf-fd2d607d497c" (UID: "f93519ae-58de-4fe9-bbdf-fd2d607d497c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.340713 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340396    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-data" pod "4b6a26be-45d7-47e9-a62e-5920e516f037" (UID: "4b6a26be-45d7-47e9-a62e-5920e516f037"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.340713 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340421    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-credential-socket" pod "f93519ae-58de-4fe9-bbdf-fd2d607d497c" (UID: "f93519ae-58de-4fe9-bbdf-fd2d607d497c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.340887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340752    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6a26be-45d7-47e9-a62e-5920e516f037-istiod-ca-cert" pod "4b6a26be-45d7-47e9-a62e-5920e516f037" (UID: "4b6a26be-45d7-47e9-a62e-5920e516f037"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:04:56.340887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340778    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.340887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340812    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.340887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340830    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.340887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340844    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.340887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340858    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.340887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340872    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.340887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340884    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.341369 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340897    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.341369 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.340974    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-certs" pod "4b6a26be-45d7-47e9-a62e-5920e516f037" (UID: "4b6a26be-45d7-47e9-a62e-5920e516f037"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.342308 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.342248    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-token" pod "f93519ae-58de-4fe9-bbdf-fd2d607d497c" (UID: "f93519ae-58de-4fe9-bbdf-fd2d607d497c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:04:56.342308 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.342249    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-podinfo" pod "f93519ae-58de-4fe9-bbdf-fd2d607d497c" (UID: "f93519ae-58de-4fe9-bbdf-fd2d607d497c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:04:56.342551 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.342349    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-envoy" pod "4b6a26be-45d7-47e9-a62e-5920e516f037" (UID: "4b6a26be-45d7-47e9-a62e-5920e516f037"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.342831 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.342808    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-token" pod "4b6a26be-45d7-47e9-a62e-5920e516f037" (UID: "4b6a26be-45d7-47e9-a62e-5920e516f037"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:04:56.343277 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.343257    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-kube-api-access-s6t2h" pod "4b6a26be-45d7-47e9-a62e-5920e516f037" (UID: "4b6a26be-45d7-47e9-a62e-5920e516f037"). InnerVolumeSpecName "kube-api-access-s6t2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:04:56.343365 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.343306    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-podinfo" pod "4b6a26be-45d7-47e9-a62e-5920e516f037" (UID: "4b6a26be-45d7-47e9-a62e-5920e516f037"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:04:56.343957 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.343937    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-kube-api-access-95d8r" pod "f93519ae-58de-4fe9-bbdf-fd2d607d497c" (UID: "f93519ae-58de-4fe9-bbdf-fd2d607d497c"). InnerVolumeSpecName "kube-api-access-95d8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:04:56.344019 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.344001    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-envoy" pod "f93519ae-58de-4fe9-bbdf-fd2d607d497c" (UID: "f93519ae-58de-4fe9-bbdf-fd2d607d497c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.442420 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.442380    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.442420 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.442416    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95d8r\" (UniqueName: \"kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-kube-api-access-95d8r\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.442420 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.442429    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6a26be-45d7-47e9-a62e-5920e516f037-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.442686 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.442439    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6a26be-45d7-47e9-a62e-5920e516f037-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.442686 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.442449    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.442686 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.442458    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.442686 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.442471    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6t2h\" (UniqueName: \"kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-kube-api-access-s6t2h\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.442686 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.442484    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6a26be-45d7-47e9-a62e-5920e516f037-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.442686 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.442493    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.442686 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.442502    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93519ae-58de-4fe9-bbdf-fd2d607d497c-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.980268 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.980240    4449 generic.go:356] "Generic (PLEG): container finished" podID="4b6a26be-45d7-47e9-a62e-5920e516f037" containerID="aa8855b994a1ed196f1597e5ff7285733bf215c8a77e395ac384ba76fccdaa9c" exitCode=0
Nov 02 10:04:56.980745 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.980365    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"
Nov 02 10:04:56.980745 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.980383    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s" event={"ID":"4b6a26be-45d7-47e9-a62e-5920e516f037","Type":"ContainerDied","Data":"aa8855b994a1ed196f1597e5ff7285733bf215c8a77e395ac384ba76fccdaa9c"}
Nov 02 10:04:56.980745 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.980451    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s" event={"ID":"4b6a26be-45d7-47e9-a62e-5920e516f037","Type":"ContainerDied","Data":"b9ad36d4ec3a38282dbd078ad0520b6e37d2d7c3ade735431e44a50033c4f8d6"}
Nov 02 10:04:56.980745 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.980483    4449 scope.go:122] "RemoveContainer" containerID="aa8855b994a1ed196f1597e5ff7285733bf215c8a77e395ac384ba76fccdaa9c"
Nov 02 10:04:56.982047 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.982022    4449 generic.go:356] "Generic (PLEG): container finished" podID="f93519ae-58de-4fe9-bbdf-fd2d607d497c" containerID="47e171145fb459186ba8bd726848b362cf44d6c4ba37888fdd4d75677015d215" exitCode=0
Nov 02 10:04:56.982186 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.982065    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"
Nov 02 10:04:56.982186 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.982064    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j" event={"ID":"f93519ae-58de-4fe9-bbdf-fd2d607d497c","Type":"ContainerDied","Data":"47e171145fb459186ba8bd726848b362cf44d6c4ba37888fdd4d75677015d215"}
Nov 02 10:04:56.982186 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.982171    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j" event={"ID":"f93519ae-58de-4fe9-bbdf-fd2d607d497c","Type":"ContainerDied","Data":"24f73c22f64a2df17b532ca51e0cff59817a8268da5206c0cd46d8c174d5c573"}
Nov 02 10:04:56.991604 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.991586    4449 scope.go:122] "RemoveContainer" containerID="aa8855b994a1ed196f1597e5ff7285733bf215c8a77e395ac384ba76fccdaa9c"
Nov 02 10:04:56.991873 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:04:56.991848    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa8855b994a1ed196f1597e5ff7285733bf215c8a77e395ac384ba76fccdaa9c\": container with ID starting with aa8855b994a1ed196f1597e5ff7285733bf215c8a77e395ac384ba76fccdaa9c not found: ID does not exist" containerID="aa8855b994a1ed196f1597e5ff7285733bf215c8a77e395ac384ba76fccdaa9c"
Nov 02 10:04:56.991949 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.991882    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa8855b994a1ed196f1597e5ff7285733bf215c8a77e395ac384ba76fccdaa9c"} err="failed to get container status \"aa8855b994a1ed196f1597e5ff7285733bf215c8a77e395ac384ba76fccdaa9c\": rpc error: code = NotFound desc = could not find container \"aa8855b994a1ed196f1597e5ff7285733bf215c8a77e395ac384ba76fccdaa9c\": container with ID starting with aa8855b994a1ed196f1597e5ff7285733bf215c8a77e395ac384ba76fccdaa9c not found: ID does not exist"
Nov 02 10:04:56.991949 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:56.991901    4449 scope.go:122] "RemoveContainer" containerID="47e171145fb459186ba8bd726848b362cf44d6c4ba37888fdd4d75677015d215"
Nov 02 10:04:57.001733 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:57.001715    4449 scope.go:122] "RemoveContainer" containerID="47e171145fb459186ba8bd726848b362cf44d6c4ba37888fdd4d75677015d215"
Nov 02 10:04:57.002021 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:04:57.001972    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e171145fb459186ba8bd726848b362cf44d6c4ba37888fdd4d75677015d215\": container with ID starting with 47e171145fb459186ba8bd726848b362cf44d6c4ba37888fdd4d75677015d215 not found: ID does not exist" containerID="47e171145fb459186ba8bd726848b362cf44d6c4ba37888fdd4d75677015d215"
Nov 02 10:04:57.002021 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:57.002001    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e171145fb459186ba8bd726848b362cf44d6c4ba37888fdd4d75677015d215"} err="failed to get container status \"47e171145fb459186ba8bd726848b362cf44d6c4ba37888fdd4d75677015d215\": rpc error: code = NotFound desc = could not find container \"47e171145fb459186ba8bd726848b362cf44d6c4ba37888fdd4d75677015d215\": container with ID starting with 47e171145fb459186ba8bd726848b362cf44d6c4ba37888fdd4d75677015d215 not found: ID does not exist"
Nov 02 10:04:57.016389 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:57.012423    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"]
Nov 02 10:04:57.020323 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:57.020303    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-fxn9s"]
Nov 02 10:04:57.032283 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:57.032258    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"]
Nov 02 10:04:57.037086 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:57.037063    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-qbv6j"]
Nov 02 10:04:57.206386 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:57.206307    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b6a26be-45d7-47e9-a62e-5920e516f037" path="/var/lib/kubelet/pods/4b6a26be-45d7-47e9-a62e-5920e516f037/volumes"
Nov 02 10:04:57.206812 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:57.206799    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93519ae-58de-4fe9-bbdf-fd2d607d497c" path="/var/lib/kubelet/pods/f93519ae-58de-4fe9-bbdf-fd2d607d497c/volumes"
Nov 02 10:04:59.588115 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:59.588081    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79" podUID="db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:04:59.588565 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:04:59.588082    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv" podUID="d679fb2c-7057-4104-9b55-3cfa4fcde09c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:05:03.027116 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:03.027083    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"]
Nov 02 10:05:03.032195 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:03.032162    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91778291-q962v"]
Nov 02 10:05:03.206550 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:03.206514    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08a49e9-6d49-4ddc-9288-f6a38b620411" path="/var/lib/kubelet/pods/b08a49e9-6d49-4ddc-9288-f6a38b620411/volumes"
Nov 02 10:05:05.688667 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:05.688625    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"]
Nov 02 10:05:05.689241 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:05.688897    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9" podUID="978d530c-fd42-4e42-80cd-746b1e37f40f" containerName="istio-proxy" containerID="cri-o://6f2d2417ddd9aa48edb2315aa07f0ed216def28593afbaa677f3d7c4d3421f22" gracePeriod=30
Nov 02 10:05:05.696967 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:05.696936    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"]
Nov 02 10:05:05.697251 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:05.697186    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb" podUID="a0a51c35-655c-4e06-8524-e19e99eb48f1" containerName="istio-proxy" containerID="cri-o://0dbb3ab956287fc15110c0cac978afe02e7804bcf581ed6ca653a43d5ba43905" gracePeriod=30
Nov 02 10:05:05.702871 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:05.702845    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"]
Nov 02 10:05:05.703111 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:05.703084    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx" podUID="a37648fb-1120-4f9d-a25b-2158bbbd3483" containerName="istio-proxy" containerID="cri-o://0a9f0a8a41322ef199e57fcd5a42b9caf424125eda136439c2e60919a95370b6" gracePeriod=30
Nov 02 10:05:05.706062 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:05.706038    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"]
Nov 02 10:05:05.706274 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:05.706253    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4" podUID="58ac13cf-aa96-4117-825a-8b4aba00c77e" containerName="istio-proxy" containerID="cri-o://6306f837eb49c430bdcb2027748be76b27bc7e57788711067a3d32ff72ce6b75" gracePeriod=30
Nov 02 10:05:06.548002 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:05:06.547966    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:05:06.748131 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:06.748093    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-795bcb67b4-xgtc6_f384c8c8-d8e1-447f-b53a-cf98f7103e28/authorino/0.log"
Nov 02 10:05:06.978329 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:06.978255    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b66fk_36151d2a-760c-4b7c-bcee-daf2d6d3a562/manager/0.log"
Nov 02 10:05:07.216779 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:07.216753    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tjlwm_11792e8e-2931-4725-aece-39791e1c90bb/manager/0.log"
Nov 02 10:05:07.401387 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:07.401357    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-52xrj_7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708/kuadrant-console-plugin/0.log"
Nov 02 10:05:08.027343 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.027317    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"
Nov 02 10:05:08.061683 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.061656    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"
Nov 02 10:05:08.070563 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.070532    4449 generic.go:356] "Generic (PLEG): container finished" podID="d679fb2c-7057-4104-9b55-3cfa4fcde09c" containerID="1040990dd13ddc3f638a1ba13e02434d80f61c72d61f006e7745505cf84469d6" exitCode=0
Nov 02 10:05:08.070748 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.070581    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"
Nov 02 10:05:08.070748 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.070616    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv" event={"ID":"d679fb2c-7057-4104-9b55-3cfa4fcde09c","Type":"ContainerDied","Data":"1040990dd13ddc3f638a1ba13e02434d80f61c72d61f006e7745505cf84469d6"}
Nov 02 10:05:08.070748 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.070715    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv" event={"ID":"d679fb2c-7057-4104-9b55-3cfa4fcde09c","Type":"ContainerDied","Data":"4800932682cc70f023add2114c16cc689a82d399c0bcfff65a1392e4b09fc3e1"}
Nov 02 10:05:08.070916 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.070750    4449 scope.go:122] "RemoveContainer" containerID="1040990dd13ddc3f638a1ba13e02434d80f61c72d61f006e7745505cf84469d6"
Nov 02 10:05:08.072342 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.072321    4449 generic.go:356] "Generic (PLEG): container finished" podID="db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f" containerID="308bb8e5c0caa410f11cf281122f6d5722f82285c5ccaacb3fdb877c7d707fb5" exitCode=0
Nov 02 10:05:08.072467 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.072373    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"
Nov 02 10:05:08.072467 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.072391    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79" event={"ID":"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f","Type":"ContainerDied","Data":"308bb8e5c0caa410f11cf281122f6d5722f82285c5ccaacb3fdb877c7d707fb5"}
Nov 02 10:05:08.072467 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.072428    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79" event={"ID":"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f","Type":"ContainerDied","Data":"59fe80233168362dbf2cfde147f8cba24a8d51e35a0e7599161c02e97e028b22"}
Nov 02 10:05:08.072680 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.072661    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d679fb2c-7057-4104-9b55-3cfa4fcde09c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d679fb2c-7057-4104-9b55-3cfa4fcde09c-plugins-config-volume\") pod \"d679fb2c-7057-4104-9b55-3cfa4fcde09c\" (UID: \"d679fb2c-7057-4104-9b55-3cfa4fcde09c\") "
Nov 02 10:05:08.072790 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.072772    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d679fb2c-7057-4104-9b55-3cfa4fcde09c-kube-api-access-zrr4s\" (UniqueName: \"kubernetes.io/projected/d679fb2c-7057-4104-9b55-3cfa4fcde09c-kube-api-access-zrr4s\") pod \"d679fb2c-7057-4104-9b55-3cfa4fcde09c\" (UID: \"d679fb2c-7057-4104-9b55-3cfa4fcde09c\") "
Nov 02 10:05:08.073033 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.073011    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d679fb2c-7057-4104-9b55-3cfa4fcde09c-plugins-config-volume" pod "d679fb2c-7057-4104-9b55-3cfa4fcde09c" (UID: "d679fb2c-7057-4104-9b55-3cfa4fcde09c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:08.073251 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.073209    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d679fb2c-7057-4104-9b55-3cfa4fcde09c-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:08.075065 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.075043    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d679fb2c-7057-4104-9b55-3cfa4fcde09c-kube-api-access-zrr4s" pod "d679fb2c-7057-4104-9b55-3cfa4fcde09c" (UID: "d679fb2c-7057-4104-9b55-3cfa4fcde09c"). InnerVolumeSpecName "kube-api-access-zrr4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:08.086704 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.086663    4449 scope.go:122] "RemoveContainer" containerID="1040990dd13ddc3f638a1ba13e02434d80f61c72d61f006e7745505cf84469d6"
Nov 02 10:05:08.087395 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:05:08.087361    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1040990dd13ddc3f638a1ba13e02434d80f61c72d61f006e7745505cf84469d6\": container with ID starting with 1040990dd13ddc3f638a1ba13e02434d80f61c72d61f006e7745505cf84469d6 not found: ID does not exist" containerID="1040990dd13ddc3f638a1ba13e02434d80f61c72d61f006e7745505cf84469d6"
Nov 02 10:05:08.087555 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.087438    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1040990dd13ddc3f638a1ba13e02434d80f61c72d61f006e7745505cf84469d6"} err="failed to get container status \"1040990dd13ddc3f638a1ba13e02434d80f61c72d61f006e7745505cf84469d6\": rpc error: code = NotFound desc = could not find container \"1040990dd13ddc3f638a1ba13e02434d80f61c72d61f006e7745505cf84469d6\": container with ID starting with 1040990dd13ddc3f638a1ba13e02434d80f61c72d61f006e7745505cf84469d6 not found: ID does not exist"
Nov 02 10:05:08.087555 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.087466    4449 scope.go:122] "RemoveContainer" containerID="308bb8e5c0caa410f11cf281122f6d5722f82285c5ccaacb3fdb877c7d707fb5"
Nov 02 10:05:08.102621 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.102594    4449 scope.go:122] "RemoveContainer" containerID="308bb8e5c0caa410f11cf281122f6d5722f82285c5ccaacb3fdb877c7d707fb5"
Nov 02 10:05:08.102997 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:05:08.102976    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308bb8e5c0caa410f11cf281122f6d5722f82285c5ccaacb3fdb877c7d707fb5\": container with ID starting with 308bb8e5c0caa410f11cf281122f6d5722f82285c5ccaacb3fdb877c7d707fb5 not found: ID does not exist" containerID="308bb8e5c0caa410f11cf281122f6d5722f82285c5ccaacb3fdb877c7d707fb5"
Nov 02 10:05:08.103064 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.103007    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308bb8e5c0caa410f11cf281122f6d5722f82285c5ccaacb3fdb877c7d707fb5"} err="failed to get container status \"308bb8e5c0caa410f11cf281122f6d5722f82285c5ccaacb3fdb877c7d707fb5\": rpc error: code = NotFound desc = could not find container \"308bb8e5c0caa410f11cf281122f6d5722f82285c5ccaacb3fdb877c7d707fb5\": container with ID starting with 308bb8e5c0caa410f11cf281122f6d5722f82285c5ccaacb3fdb877c7d707fb5 not found: ID does not exist"
Nov 02 10:05:08.174076 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.174040    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-kube-api-access-czfwz\" (UniqueName: \"kubernetes.io/projected/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-kube-api-access-czfwz\") pod \"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f\" (UID: \"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f\") "
Nov 02 10:05:08.174316 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.174180    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-plugins-config-volume\") pod \"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f\" (UID: \"db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f\") "
Nov 02 10:05:08.174463 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.174448    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrr4s\" (UniqueName: \"kubernetes.io/projected/d679fb2c-7057-4104-9b55-3cfa4fcde09c-kube-api-access-zrr4s\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:08.174615 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.174591    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-plugins-config-volume" pod "db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f" (UID: "db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:08.176375 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.176354    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-kube-api-access-czfwz" pod "db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f" (UID: "db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f"). InnerVolumeSpecName "kube-api-access-czfwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:08.276113 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.276020    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czfwz\" (UniqueName: \"kubernetes.io/projected/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-kube-api-access-czfwz\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:08.276113 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.276057    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:08.362086 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.362041    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2l8n5_f719b895-f939-4354-b42a-9c252a5f7712/manager/0.log"
Nov 02 10:05:08.401655 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.401619    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"]
Nov 02 10:05:08.405548 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.405520    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-629129-ddf565c67-76b79"]
Nov 02 10:05:08.419227 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.419181    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"]
Nov 02 10:05:08.424147 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:08.424114    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-629129-9d8b9d665-qclfv"]
Nov 02 10:05:09.206684 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:09.206647    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d679fb2c-7057-4104-9b55-3cfa4fcde09c" path="/var/lib/kubelet/pods/d679fb2c-7057-4104-9b55-3cfa4fcde09c/volumes"
Nov 02 10:05:09.207256 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:09.207237    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f" path="/var/lib/kubelet/pods/db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f/volumes"
Nov 02 10:05:09.766253 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:09.766225    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cwjgx_1cc56635-aa7a-4702-8575-2c73011a580f/manager/0.log"
Nov 02 10:05:11.003203 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.003169    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 10:05:11.011590 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.011563    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 10:05:11.056524 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.055860    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 10:05:11.063362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.063341    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:05:11.095727 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.095698    4449 generic.go:356] "Generic (PLEG): container finished" podID="a0a51c35-655c-4e06-8524-e19e99eb48f1" containerID="0dbb3ab956287fc15110c0cac978afe02e7804bcf581ed6ca653a43d5ba43905" exitCode=0
Nov 02 10:05:11.095915 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.095744    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"
Nov 02 10:05:11.095915 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.095780    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb" event={"ID":"a0a51c35-655c-4e06-8524-e19e99eb48f1","Type":"ContainerDied","Data":"0dbb3ab956287fc15110c0cac978afe02e7804bcf581ed6ca653a43d5ba43905"}
Nov 02 10:05:11.095915 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.095837    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb" event={"ID":"a0a51c35-655c-4e06-8524-e19e99eb48f1","Type":"ContainerDied","Data":"e418bcb4bb90fe10fd957308f20a97aafcc860103525dd2af573523116862910"}
Nov 02 10:05:11.095915 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.095868    4449 scope.go:122] "RemoveContainer" containerID="0dbb3ab956287fc15110c0cac978afe02e7804bcf581ed6ca653a43d5ba43905"
Nov 02 10:05:11.097205 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.097182    4449 generic.go:356] "Generic (PLEG): container finished" podID="978d530c-fd42-4e42-80cd-746b1e37f40f" containerID="6f2d2417ddd9aa48edb2315aa07f0ed216def28593afbaa677f3d7c4d3421f22" exitCode=0
Nov 02 10:05:11.097328 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.097253    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9" event={"ID":"978d530c-fd42-4e42-80cd-746b1e37f40f","Type":"ContainerDied","Data":"6f2d2417ddd9aa48edb2315aa07f0ed216def28593afbaa677f3d7c4d3421f22"}
Nov 02 10:05:11.097328 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.097298    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9" event={"ID":"978d530c-fd42-4e42-80cd-746b1e37f40f","Type":"ContainerDied","Data":"ccd128c50a4bc0b2349d0248216b1e8f8446f4fb200a6552739c5040fd808247"}
Nov 02 10:05:11.097631 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.097259    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"
Nov 02 10:05:11.098867 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.098842    4449 generic.go:356] "Generic (PLEG): container finished" podID="a37648fb-1120-4f9d-a25b-2158bbbd3483" containerID="0a9f0a8a41322ef199e57fcd5a42b9caf424125eda136439c2e60919a95370b6" exitCode=0
Nov 02 10:05:11.098986 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.098886    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"
Nov 02 10:05:11.098986 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.098892    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx" event={"ID":"a37648fb-1120-4f9d-a25b-2158bbbd3483","Type":"ContainerDied","Data":"0a9f0a8a41322ef199e57fcd5a42b9caf424125eda136439c2e60919a95370b6"}
Nov 02 10:05:11.098986 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.098935    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx" event={"ID":"a37648fb-1120-4f9d-a25b-2158bbbd3483","Type":"ContainerDied","Data":"701ec9a71a62db5e28e8f11bd34ac78e6e1480d108210516d37a04d0c3a9df18"}
Nov 02 10:05:11.100501 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.100480    4449 generic.go:356] "Generic (PLEG): container finished" podID="58ac13cf-aa96-4117-825a-8b4aba00c77e" containerID="6306f837eb49c430bdcb2027748be76b27bc7e57788711067a3d32ff72ce6b75" exitCode=0
Nov 02 10:05:11.100598 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.100523    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"
Nov 02 10:05:11.100659 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.100524    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4" event={"ID":"58ac13cf-aa96-4117-825a-8b4aba00c77e","Type":"ContainerDied","Data":"6306f837eb49c430bdcb2027748be76b27bc7e57788711067a3d32ff72ce6b75"}
Nov 02 10:05:11.100659 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.100637    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4" event={"ID":"58ac13cf-aa96-4117-825a-8b4aba00c77e","Type":"ContainerDied","Data":"f82f15d799df41713eb0c3cea2a1d59f864ab766d8774ac67710942b104f4caa"}
Nov 02 10:05:11.106756 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.106740    4449 scope.go:122] "RemoveContainer" containerID="0dbb3ab956287fc15110c0cac978afe02e7804bcf581ed6ca653a43d5ba43905"
Nov 02 10:05:11.106854 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.106835    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-certs\") pod \"a37648fb-1120-4f9d-a25b-2158bbbd3483\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") "
Nov 02 10:05:11.106914 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.106878    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-podinfo\") pod \"978d530c-fd42-4e42-80cd-746b1e37f40f\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") "
Nov 02 10:05:11.106977 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.106911    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-credential-socket\") pod \"a0a51c35-655c-4e06-8524-e19e99eb48f1\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") "
Nov 02 10:05:11.106977 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.106942    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-envoy\") pod \"58ac13cf-aa96-4117-825a-8b4aba00c77e\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") "
Nov 02 10:05:11.107352 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107323    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-credential-socket" pod "a0a51c35-655c-4e06-8524-e19e99eb48f1" (UID: "a0a51c35-655c-4e06-8524-e19e99eb48f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.107468 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:05:11.107417    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbb3ab956287fc15110c0cac978afe02e7804bcf581ed6ca653a43d5ba43905\": container with ID starting with 0dbb3ab956287fc15110c0cac978afe02e7804bcf581ed6ca653a43d5ba43905 not found: ID does not exist" containerID="0dbb3ab956287fc15110c0cac978afe02e7804bcf581ed6ca653a43d5ba43905"
Nov 02 10:05:11.107468 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107449    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbb3ab956287fc15110c0cac978afe02e7804bcf581ed6ca653a43d5ba43905"} err="failed to get container status \"0dbb3ab956287fc15110c0cac978afe02e7804bcf581ed6ca653a43d5ba43905\": rpc error: code = NotFound desc = could not find container \"0dbb3ab956287fc15110c0cac978afe02e7804bcf581ed6ca653a43d5ba43905\": container with ID starting with 0dbb3ab956287fc15110c0cac978afe02e7804bcf581ed6ca653a43d5ba43905 not found: ID does not exist"
Nov 02 10:05:11.107605 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107469    4449 scope.go:122] "RemoveContainer" containerID="6f2d2417ddd9aa48edb2315aa07f0ed216def28593afbaa677f3d7c4d3421f22"
Nov 02 10:05:11.107605 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107504    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-certs" pod "a37648fb-1120-4f9d-a25b-2158bbbd3483" (UID: "a37648fb-1120-4f9d-a25b-2158bbbd3483"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.107605 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107555    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-socket\") pod \"978d530c-fd42-4e42-80cd-746b1e37f40f\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") "
Nov 02 10:05:11.107770 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107614    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-certs\") pod \"978d530c-fd42-4e42-80cd-746b1e37f40f\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") "
Nov 02 10:05:11.107770 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107651    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-podinfo\") pod \"a37648fb-1120-4f9d-a25b-2158bbbd3483\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") "
Nov 02 10:05:11.107770 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107684    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-certs\") pod \"a0a51c35-655c-4e06-8524-e19e99eb48f1\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") "
Nov 02 10:05:11.107770 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107712    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-credential-socket\") pod \"a37648fb-1120-4f9d-a25b-2158bbbd3483\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") "
Nov 02 10:05:11.107770 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107741    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-data\") pod \"a37648fb-1120-4f9d-a25b-2158bbbd3483\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") "
Nov 02 10:05:11.108032 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107774    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-envoy\") pod \"978d530c-fd42-4e42-80cd-746b1e37f40f\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") "
Nov 02 10:05:11.108032 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107809    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-kube-api-access-kxf9r\" (UniqueName: \"kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-kube-api-access-kxf9r\") pod \"a0a51c35-655c-4e06-8524-e19e99eb48f1\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") "
Nov 02 10:05:11.108032 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107848    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-data\") pod \"978d530c-fd42-4e42-80cd-746b1e37f40f\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") "
Nov 02 10:05:11.108032 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107894    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-socket\") pod \"a37648fb-1120-4f9d-a25b-2158bbbd3483\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") "
Nov 02 10:05:11.108032 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.107941    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-podinfo\") pod \"a0a51c35-655c-4e06-8524-e19e99eb48f1\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") "
Nov 02 10:05:11.108032 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108017    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-data\") pod \"a0a51c35-655c-4e06-8524-e19e99eb48f1\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") "
Nov 02 10:05:11.108380 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108054    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-certs" pod "a0a51c35-655c-4e06-8524-e19e99eb48f1" (UID: "a0a51c35-655c-4e06-8524-e19e99eb48f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.108380 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108059    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-envoy\") pod \"a37648fb-1120-4f9d-a25b-2158bbbd3483\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") "
Nov 02 10:05:11.108380 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108112    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-certs" pod "978d530c-fd42-4e42-80cd-746b1e37f40f" (UID: "978d530c-fd42-4e42-80cd-746b1e37f40f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.108380 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108169    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-credential-socket\") pod \"978d530c-fd42-4e42-80cd-746b1e37f40f\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") "
Nov 02 10:05:11.108380 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108209    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-token\" (UniqueName: \"kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-token\") pod \"a0a51c35-655c-4e06-8524-e19e99eb48f1\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") "
Nov 02 10:05:11.108380 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108272    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-socket\") pod \"a0a51c35-655c-4e06-8524-e19e99eb48f1\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") "
Nov 02 10:05:11.108380 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108311    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-envoy\") pod \"a0a51c35-655c-4e06-8524-e19e99eb48f1\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") "
Nov 02 10:05:11.108380 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108325    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-socket" pod "978d530c-fd42-4e42-80cd-746b1e37f40f" (UID: "978d530c-fd42-4e42-80cd-746b1e37f40f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.108380 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108344    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-kube-api-access-dsxpx\" (UniqueName: \"kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-kube-api-access-dsxpx\") pod \"a37648fb-1120-4f9d-a25b-2158bbbd3483\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") "
Nov 02 10:05:11.108380 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108384    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-token\" (UniqueName: \"kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-token\") pod \"a37648fb-1120-4f9d-a25b-2158bbbd3483\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") "
Nov 02 10:05:11.109096 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108425    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-podinfo\") pod \"58ac13cf-aa96-4117-825a-8b4aba00c77e\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") "
Nov 02 10:05:11.109096 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108463    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978d530c-fd42-4e42-80cd-746b1e37f40f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978d530c-fd42-4e42-80cd-746b1e37f40f-istiod-ca-cert\") pod \"978d530c-fd42-4e42-80cd-746b1e37f40f\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") "
Nov 02 10:05:11.109096 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108492    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a51c35-655c-4e06-8524-e19e99eb48f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a51c35-655c-4e06-8524-e19e99eb48f1-istiod-ca-cert\") pod \"a0a51c35-655c-4e06-8524-e19e99eb48f1\" (UID: \"a0a51c35-655c-4e06-8524-e19e99eb48f1\") "
Nov 02 10:05:11.109096 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108532    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-socket\") pod \"58ac13cf-aa96-4117-825a-8b4aba00c77e\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") "
Nov 02 10:05:11.109096 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108564    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-certs\") pod \"58ac13cf-aa96-4117-825a-8b4aba00c77e\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") "
Nov 02 10:05:11.109096 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108594    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-credential-socket\") pod \"58ac13cf-aa96-4117-825a-8b4aba00c77e\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") "
Nov 02 10:05:11.109096 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108624    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37648fb-1120-4f9d-a25b-2158bbbd3483-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37648fb-1120-4f9d-a25b-2158bbbd3483-istiod-ca-cert\") pod \"a37648fb-1120-4f9d-a25b-2158bbbd3483\" (UID: \"a37648fb-1120-4f9d-a25b-2158bbbd3483\") "
Nov 02 10:05:11.109096 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108651    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-kube-api-access-d9blh\" (UniqueName: \"kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-kube-api-access-d9blh\") pod \"978d530c-fd42-4e42-80cd-746b1e37f40f\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") "
Nov 02 10:05:11.109096 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108736    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-kube-api-access-xl6h4\" (UniqueName: \"kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-kube-api-access-xl6h4\") pod \"58ac13cf-aa96-4117-825a-8b4aba00c77e\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") "
Nov 02 10:05:11.109096 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108770    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-data\") pod \"58ac13cf-aa96-4117-825a-8b4aba00c77e\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") "
Nov 02 10:05:11.109096 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.108834    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-token\" (UniqueName: \"kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-token\") pod \"58ac13cf-aa96-4117-825a-8b4aba00c77e\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") "
Nov 02 10:05:11.109742 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.109194    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ac13cf-aa96-4117-825a-8b4aba00c77e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ac13cf-aa96-4117-825a-8b4aba00c77e-istiod-ca-cert\") pod \"58ac13cf-aa96-4117-825a-8b4aba00c77e\" (UID: \"58ac13cf-aa96-4117-825a-8b4aba00c77e\") "
Nov 02 10:05:11.109742 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.109255    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-token\" (UniqueName: \"kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-token\") pod \"978d530c-fd42-4e42-80cd-746b1e37f40f\" (UID: \"978d530c-fd42-4e42-80cd-746b1e37f40f\") "
Nov 02 10:05:11.109854 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.109791    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.109854 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.109814    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.109854 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.109830    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.109854 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.109844    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.110077 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.109859    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.110886 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.110229    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-data" pod "978d530c-fd42-4e42-80cd-746b1e37f40f" (UID: "978d530c-fd42-4e42-80cd-746b1e37f40f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.110886 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.110459    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-data" pod "a37648fb-1120-4f9d-a25b-2158bbbd3483" (UID: "a37648fb-1120-4f9d-a25b-2158bbbd3483"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.110886 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.110497    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-socket" pod "a37648fb-1120-4f9d-a25b-2158bbbd3483" (UID: "a37648fb-1120-4f9d-a25b-2158bbbd3483"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.110886 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.110668    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-envoy" pod "58ac13cf-aa96-4117-825a-8b4aba00c77e" (UID: "58ac13cf-aa96-4117-825a-8b4aba00c77e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.110886 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.110700    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-credential-socket" pod "a37648fb-1120-4f9d-a25b-2158bbbd3483" (UID: "a37648fb-1120-4f9d-a25b-2158bbbd3483"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.111373 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.110893    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-data" pod "a0a51c35-655c-4e06-8524-e19e99eb48f1" (UID: "a0a51c35-655c-4e06-8524-e19e99eb48f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.111373 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.111159    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a51c35-655c-4e06-8524-e19e99eb48f1-istiod-ca-cert" pod "a0a51c35-655c-4e06-8524-e19e99eb48f1" (UID: "a0a51c35-655c-4e06-8524-e19e99eb48f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:11.115194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.114180    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-certs" pod "58ac13cf-aa96-4117-825a-8b4aba00c77e" (UID: "58ac13cf-aa96-4117-825a-8b4aba00c77e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.115194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.114731    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37648fb-1120-4f9d-a25b-2158bbbd3483-istiod-ca-cert" pod "a37648fb-1120-4f9d-a25b-2158bbbd3483" (UID: "a37648fb-1120-4f9d-a25b-2158bbbd3483"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:11.115194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.114781    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-credential-socket" pod "58ac13cf-aa96-4117-825a-8b4aba00c77e" (UID: "58ac13cf-aa96-4117-825a-8b4aba00c77e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.115194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.114815    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-socket" pod "58ac13cf-aa96-4117-825a-8b4aba00c77e" (UID: "58ac13cf-aa96-4117-825a-8b4aba00c77e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.115627 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.115522    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-data" pod "58ac13cf-aa96-4117-825a-8b4aba00c77e" (UID: "58ac13cf-aa96-4117-825a-8b4aba00c77e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.116123 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.116099    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-envoy" pod "a37648fb-1120-4f9d-a25b-2158bbbd3483" (UID: "a37648fb-1120-4f9d-a25b-2158bbbd3483"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.117269 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.116479    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ac13cf-aa96-4117-825a-8b4aba00c77e-istiod-ca-cert" pod "58ac13cf-aa96-4117-825a-8b4aba00c77e" (UID: "58ac13cf-aa96-4117-825a-8b4aba00c77e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:11.117403 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.117122    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978d530c-fd42-4e42-80cd-746b1e37f40f-istiod-ca-cert" pod "978d530c-fd42-4e42-80cd-746b1e37f40f" (UID: "978d530c-fd42-4e42-80cd-746b1e37f40f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:11.117790 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.117573    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-socket" pod "a0a51c35-655c-4e06-8524-e19e99eb48f1" (UID: "a0a51c35-655c-4e06-8524-e19e99eb48f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.117790 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.117693    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-credential-socket" pod "978d530c-fd42-4e42-80cd-746b1e37f40f" (UID: "978d530c-fd42-4e42-80cd-746b1e37f40f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.120465 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.120425    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-envoy" pod "978d530c-fd42-4e42-80cd-746b1e37f40f" (UID: "978d530c-fd42-4e42-80cd-746b1e37f40f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.120956 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.120930    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-token" pod "978d530c-fd42-4e42-80cd-746b1e37f40f" (UID: "978d530c-fd42-4e42-80cd-746b1e37f40f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:11.121067 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.121013    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-kube-api-access-d9blh" pod "978d530c-fd42-4e42-80cd-746b1e37f40f" (UID: "978d530c-fd42-4e42-80cd-746b1e37f40f"). InnerVolumeSpecName "kube-api-access-d9blh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:11.121143 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.121076    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-podinfo" pod "a0a51c35-655c-4e06-8524-e19e99eb48f1" (UID: "a0a51c35-655c-4e06-8524-e19e99eb48f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:05:11.121200 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.121170    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-token" pod "a37648fb-1120-4f9d-a25b-2158bbbd3483" (UID: "a37648fb-1120-4f9d-a25b-2158bbbd3483"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:11.121301 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.121261    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-envoy" pod "a0a51c35-655c-4e06-8524-e19e99eb48f1" (UID: "a0a51c35-655c-4e06-8524-e19e99eb48f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.121822 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.121560    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-token" pod "58ac13cf-aa96-4117-825a-8b4aba00c77e" (UID: "58ac13cf-aa96-4117-825a-8b4aba00c77e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:11.121822 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.121601    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-kube-api-access-kxf9r" pod "a0a51c35-655c-4e06-8524-e19e99eb48f1" (UID: "a0a51c35-655c-4e06-8524-e19e99eb48f1"). InnerVolumeSpecName "kube-api-access-kxf9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:11.121822 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.121599    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-podinfo" pod "978d530c-fd42-4e42-80cd-746b1e37f40f" (UID: "978d530c-fd42-4e42-80cd-746b1e37f40f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:05:11.123972 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.123944    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-token" pod "a0a51c35-655c-4e06-8524-e19e99eb48f1" (UID: "a0a51c35-655c-4e06-8524-e19e99eb48f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:11.124055 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.123990    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-kube-api-access-dsxpx" pod "a37648fb-1120-4f9d-a25b-2158bbbd3483" (UID: "a37648fb-1120-4f9d-a25b-2158bbbd3483"). InnerVolumeSpecName "kube-api-access-dsxpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:11.124120 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.124053    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-kube-api-access-xl6h4" pod "58ac13cf-aa96-4117-825a-8b4aba00c77e" (UID: "58ac13cf-aa96-4117-825a-8b4aba00c77e"). InnerVolumeSpecName "kube-api-access-xl6h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:11.130807 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.130774    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-podinfo" pod "58ac13cf-aa96-4117-825a-8b4aba00c77e" (UID: "58ac13cf-aa96-4117-825a-8b4aba00c77e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:05:11.132085 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.132065    4449 scope.go:122] "RemoveContainer" containerID="6f2d2417ddd9aa48edb2315aa07f0ed216def28593afbaa677f3d7c4d3421f22"
Nov 02 10:05:11.132515 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:05:11.132491    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2d2417ddd9aa48edb2315aa07f0ed216def28593afbaa677f3d7c4d3421f22\": container with ID starting with 6f2d2417ddd9aa48edb2315aa07f0ed216def28593afbaa677f3d7c4d3421f22 not found: ID does not exist" containerID="6f2d2417ddd9aa48edb2315aa07f0ed216def28593afbaa677f3d7c4d3421f22"
Nov 02 10:05:11.132614 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.132529    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2d2417ddd9aa48edb2315aa07f0ed216def28593afbaa677f3d7c4d3421f22"} err="failed to get container status \"6f2d2417ddd9aa48edb2315aa07f0ed216def28593afbaa677f3d7c4d3421f22\": rpc error: code = NotFound desc = could not find container \"6f2d2417ddd9aa48edb2315aa07f0ed216def28593afbaa677f3d7c4d3421f22\": container with ID starting with 6f2d2417ddd9aa48edb2315aa07f0ed216def28593afbaa677f3d7c4d3421f22 not found: ID does not exist"
Nov 02 10:05:11.132614 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.132563    4449 scope.go:122] "RemoveContainer" containerID="0a9f0a8a41322ef199e57fcd5a42b9caf424125eda136439c2e60919a95370b6"
Nov 02 10:05:11.135243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.135181    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-podinfo" pod "a37648fb-1120-4f9d-a25b-2158bbbd3483" (UID: "a37648fb-1120-4f9d-a25b-2158bbbd3483"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:05:11.143691 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.143670    4449 scope.go:122] "RemoveContainer" containerID="0a9f0a8a41322ef199e57fcd5a42b9caf424125eda136439c2e60919a95370b6"
Nov 02 10:05:11.144057 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:05:11.144036    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9f0a8a41322ef199e57fcd5a42b9caf424125eda136439c2e60919a95370b6\": container with ID starting with 0a9f0a8a41322ef199e57fcd5a42b9caf424125eda136439c2e60919a95370b6 not found: ID does not exist" containerID="0a9f0a8a41322ef199e57fcd5a42b9caf424125eda136439c2e60919a95370b6"
Nov 02 10:05:11.144201 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.144071    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9f0a8a41322ef199e57fcd5a42b9caf424125eda136439c2e60919a95370b6"} err="failed to get container status \"0a9f0a8a41322ef199e57fcd5a42b9caf424125eda136439c2e60919a95370b6\": rpc error: code = NotFound desc = could not find container \"0a9f0a8a41322ef199e57fcd5a42b9caf424125eda136439c2e60919a95370b6\": container with ID starting with 0a9f0a8a41322ef199e57fcd5a42b9caf424125eda136439c2e60919a95370b6 not found: ID does not exist"
Nov 02 10:05:11.144201 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.144098    4449 scope.go:122] "RemoveContainer" containerID="6306f837eb49c430bdcb2027748be76b27bc7e57788711067a3d32ff72ce6b75"
Nov 02 10:05:11.153973 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.153954    4449 scope.go:122] "RemoveContainer" containerID="6306f837eb49c430bdcb2027748be76b27bc7e57788711067a3d32ff72ce6b75"
Nov 02 10:05:11.154246 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:05:11.154224    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6306f837eb49c430bdcb2027748be76b27bc7e57788711067a3d32ff72ce6b75\": container with ID starting with 6306f837eb49c430bdcb2027748be76b27bc7e57788711067a3d32ff72ce6b75 not found: ID does not exist" containerID="6306f837eb49c430bdcb2027748be76b27bc7e57788711067a3d32ff72ce6b75"
Nov 02 10:05:11.154322 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.154254    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6306f837eb49c430bdcb2027748be76b27bc7e57788711067a3d32ff72ce6b75"} err="failed to get container status \"6306f837eb49c430bdcb2027748be76b27bc7e57788711067a3d32ff72ce6b75\": rpc error: code = NotFound desc = could not find container \"6306f837eb49c430bdcb2027748be76b27bc7e57788711067a3d32ff72ce6b75\": container with ID starting with 6306f837eb49c430bdcb2027748be76b27bc7e57788711067a3d32ff72ce6b75 not found: ID does not exist"
Nov 02 10:05:11.210549 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210520    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210549 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210547    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210557    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210565    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210574    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210583    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxf9r\" (UniqueName: \"kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-kube-api-access-kxf9r\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210592    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210601    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210609    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210618    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210626    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210634    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978d530c-fd42-4e42-80cd-746b1e37f40f-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210641    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210649    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210658    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a51c35-655c-4e06-8524-e19e99eb48f1-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210666    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsxpx\" (UniqueName: \"kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-kube-api-access-dsxpx\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210673    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37648fb-1120-4f9d-a25b-2158bbbd3483-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210681    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210689    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978d530c-fd42-4e42-80cd-746b1e37f40f-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210697    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a51c35-655c-4e06-8524-e19e99eb48f1-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210705    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210713    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210721    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210729    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37648fb-1120-4f9d-a25b-2158bbbd3483-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210738    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9blh\" (UniqueName: \"kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-kube-api-access-d9blh\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210746    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6h4\" (UniqueName: \"kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-kube-api-access-xl6h4\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.210735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210755    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.212025 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210763    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ac13cf-aa96-4117-825a-8b4aba00c77e-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.212025 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210772    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ac13cf-aa96-4117-825a-8b4aba00c77e-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.212025 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210780    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.212025 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.210788    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978d530c-fd42-4e42-80cd-746b1e37f40f-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.416847 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.416804    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"]
Nov 02 10:05:11.425525 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.425493    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb"]
Nov 02 10:05:11.442021 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.441977    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"]
Nov 02 10:05:11.446760 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.446729    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r5zlx"]
Nov 02 10:05:11.462525 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.462489    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"]
Nov 02 10:05:11.466722 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.466694    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-jqhv4"]
Nov 02 10:05:11.479185 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.479141    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"]
Nov 02 10:05:11.483433 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.483405    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-qvxr9"]
Nov 02 10:05:11.987588 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.987339    4449 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:05:11.987588 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:11.987429    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-5msfb" podUID="a0a51c35-655c-4e06-8524-e19e99eb48f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:05:13.207116 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:13.207080    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ac13cf-aa96-4117-825a-8b4aba00c77e" path="/var/lib/kubelet/pods/58ac13cf-aa96-4117-825a-8b4aba00c77e/volumes"
Nov 02 10:05:13.207560 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:13.207545    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978d530c-fd42-4e42-80cd-746b1e37f40f" path="/var/lib/kubelet/pods/978d530c-fd42-4e42-80cd-746b1e37f40f/volumes"
Nov 02 10:05:13.207938 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:13.207926    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a51c35-655c-4e06-8524-e19e99eb48f1" path="/var/lib/kubelet/pods/a0a51c35-655c-4e06-8524-e19e99eb48f1/volumes"
Nov 02 10:05:13.208623 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:13.208607    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37648fb-1120-4f9d-a25b-2158bbbd3483" path="/var/lib/kubelet/pods/a37648fb-1120-4f9d-a25b-2158bbbd3483/volumes"
Nov 02 10:05:15.823252 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.823200    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"]
Nov 02 10:05:15.823860 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.823838    4449 cpu_manager.go:414] "containerMap: removing container" podUID="4b6a26be-45d7-47e9-a62e-5920e516f037" containerName="istio-proxy"
Nov 02 10:05:15.823950 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.823862    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6a26be-45d7-47e9-a62e-5920e516f037" containerName="istio-proxy"
Nov 02 10:05:15.823950 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.823886    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a37648fb-1120-4f9d-a25b-2158bbbd3483" containerName="istio-proxy"
Nov 02 10:05:15.823950 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.823893    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37648fb-1120-4f9d-a25b-2158bbbd3483" containerName="istio-proxy"
Nov 02 10:05:15.823950 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.823912    4449 cpu_manager.go:414] "containerMap: removing container" podUID="58ac13cf-aa96-4117-825a-8b4aba00c77e" containerName="istio-proxy"
Nov 02 10:05:15.823950 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.823919    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ac13cf-aa96-4117-825a-8b4aba00c77e" containerName="istio-proxy"
Nov 02 10:05:15.823950 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.823949    4449 cpu_manager.go:414] "containerMap: removing container" podUID="d679fb2c-7057-4104-9b55-3cfa4fcde09c" containerName="payload-processing"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.823956    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d679fb2c-7057-4104-9b55-3cfa4fcde09c" containerName="payload-processing"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.823974    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f93519ae-58de-4fe9-bbdf-fd2d607d497c" containerName="istio-proxy"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.823982    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93519ae-58de-4fe9-bbdf-fd2d607d497c" containerName="istio-proxy"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.823993    4449 cpu_manager.go:414] "containerMap: removing container" podUID="1cc0e764-96e1-46cf-a580-96253011d988" containerName="revoke-keys"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824001    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc0e764-96e1-46cf-a580-96253011d988" containerName="revoke-keys"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824010    4449 cpu_manager.go:414] "containerMap: removing container" podUID="978d530c-fd42-4e42-80cd-746b1e37f40f" containerName="istio-proxy"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824017    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978d530c-fd42-4e42-80cd-746b1e37f40f" containerName="istio-proxy"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824027    4449 cpu_manager.go:414] "containerMap: removing container" podUID="db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f" containerName="payload-pre-processing"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824034    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f" containerName="payload-pre-processing"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824049    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a0a51c35-655c-4e06-8524-e19e99eb48f1" containerName="istio-proxy"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824058    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a51c35-655c-4e06-8524-e19e99eb48f1" containerName="istio-proxy"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824163    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="d679fb2c-7057-4104-9b55-3cfa4fcde09c" containerName="payload-processing"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824179    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93519ae-58de-4fe9-bbdf-fd2d607d497c" containerName="istio-proxy"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824189    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc0e764-96e1-46cf-a580-96253011d988" containerName="revoke-keys"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824198    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="978d530c-fd42-4e42-80cd-746b1e37f40f" containerName="istio-proxy"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824208    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="db6bc134-cd0f-42b5-b77b-0f0eb9fc2b2f" containerName="payload-pre-processing"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824238    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a51c35-655c-4e06-8524-e19e99eb48f1" containerName="istio-proxy"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824250    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6a26be-45d7-47e9-a62e-5920e516f037" containerName="istio-proxy"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824258    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37648fb-1120-4f9d-a25b-2158bbbd3483" containerName="istio-proxy"
Nov 02 10:05:15.824315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.824268    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ac13cf-aa96-4117-825a-8b4aba00c77e" containerName="istio-proxy"
Nov 02 10:05:15.827461 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.827437    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:15.834927 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.834739    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b9ae0feb-openshift-default-dockercfg-c2cvn\""
Nov 02 10:05:15.842254 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.842196    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"]
Nov 02 10:05:15.965072 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.965028    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-data\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:15.965333 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.965158    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-certs\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:15.965333 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.965247    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991fccf4-8422-4474-97ec-a0cb5b407788-istio-podinfo\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:15.965333 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.965308    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-socket\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:15.965530 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.965403    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-istio-token\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:15.965530 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.965440    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfktj\" (UniqueName: \"kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-kube-api-access-mfktj\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:15.965530 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.965465    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991fccf4-8422-4474-97ec-a0cb5b407788-istiod-ca-cert\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:15.965530 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.965489    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-envoy\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:15.965530 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:15.965525    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-credential-socket\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.066797 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.066762    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-socket\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.066983 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.066832    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-istio-token\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.066983 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.066857    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfktj\" (UniqueName: \"kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-kube-api-access-mfktj\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.066983 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.066876    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991fccf4-8422-4474-97ec-a0cb5b407788-istiod-ca-cert\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.066983 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.066891    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-envoy\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.066983 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.066912    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-credential-socket\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.066983 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.066956    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-data\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.067351 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.067004    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-certs\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.067351 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.067051    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991fccf4-8422-4474-97ec-a0cb5b407788-istio-podinfo\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.067351 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.067234    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-socket\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.067471 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.067438    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-credential-socket\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.067523 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.067495    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-certs\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.067580 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.067553    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-data\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.067703 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.067684    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991fccf4-8422-4474-97ec-a0cb5b407788-istiod-ca-cert\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.069412 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.069385    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-envoy\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.069640 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.069624    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991fccf4-8422-4474-97ec-a0cb5b407788-istio-podinfo\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.080161 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.080102    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfktj\" (UniqueName: \"kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-kube-api-access-mfktj\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.080161 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.080102    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-istio-token\") pod \"e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") " pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.142748 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.142715    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:16.434128 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.434095    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"]
Nov 02 10:05:16.435064 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:05:16.435042    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991fccf4_8422_4474_97ec_a0cb5b407788.slice/crio-b0875fbd5cca0575980e49d56079dead953dd5b83d6260057c52e74dbc2be546 WatchSource:0}: Error finding container b0875fbd5cca0575980e49d56079dead953dd5b83d6260057c52e74dbc2be546: Status 404 returned error can't find the container with id b0875fbd5cca0575980e49d56079dead953dd5b83d6260057c52e74dbc2be546
Nov 02 10:05:16.437389 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.437354    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:05:16.437484 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.437420    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:05:16.437484 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:16.437451    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:05:17.146179 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:17.146140    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" event={"ID":"991fccf4-8422-4474-97ec-a0cb5b407788","Type":"ContainerStarted","Data":"16e087329a092cb4110b178b172a1f1a0320243c85985995cb1d9d0a3e81c2d9"}
Nov 02 10:05:17.146659 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:17.146198    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" event={"ID":"991fccf4-8422-4474-97ec-a0cb5b407788","Type":"ContainerStarted","Data":"b0875fbd5cca0575980e49d56079dead953dd5b83d6260057c52e74dbc2be546"}
Nov 02 10:05:17.173408 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:17.173347    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" podStartSLOduration=2.173330439 podStartE2EDuration="2.173330439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:05:17.170738502 +0000 UTC m=+1794.109025289" watchObservedRunningTime="2026-11-02 10:05:17.173330439 +0000 UTC m=+1794.111617216"
Nov 02 10:05:18.143287 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:18.143252    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:18.148835 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:18.148808    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:18.153185 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:18.153164    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:18.154006 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:18.153987    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:25.913348 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:25.913312    4449 scope.go:122] "RemoveContainer" containerID="5e41a3a1ce260d51a717aab97bf47a5c55685c329a42b23c315ce626e6fb9396"
Nov 02 10:05:25.923103 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:25.923080    4449 scope.go:122] "RemoveContainer" containerID="85841e3b072864c9d711b1b4d0a1034e3a20465340e1e228e02cb73ccf638afd"
Nov 02 10:05:25.931811 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:25.931791    4449 scope.go:122] "RemoveContainer" containerID="e1c8add17351a72c6f5d1e22e8c1d8d3778cf28b8597834a626df76424a727ab"
Nov 02 10:05:25.940083 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:25.940062    4449 scope.go:122] "RemoveContainer" containerID="a2ad9b0a1d0c1f2d06ea9d26bd730a3adc6e3fcdf16326b93a902c838d599b10"
Nov 02 10:05:33.155147 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:33.155108    4449 patch_prober.go:29] interesting pod/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Nov 02 10:05:33.155624 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:33.155168    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Nov 02 10:05:33.260937 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:33.260901    4449 generic.go:356] "Generic (PLEG): container finished" podID="991fccf4-8422-4474-97ec-a0cb5b407788" containerID="16e087329a092cb4110b178b172a1f1a0320243c85985995cb1d9d0a3e81c2d9" exitCode=0
Nov 02 10:05:33.261126 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:33.260976    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" event={"ID":"991fccf4-8422-4474-97ec-a0cb5b407788","Type":"ContainerDied","Data":"16e087329a092cb4110b178b172a1f1a0320243c85985995cb1d9d0a3e81c2d9"}
Nov 02 10:05:33.261449 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:33.261432    4449 scope.go:122] "RemoveContainer" containerID="16e087329a092cb4110b178b172a1f1a0320243c85985995cb1d9d0a3e81c2d9"
Nov 02 10:05:33.262715 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:33.262680    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:05:33.262814 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:33.262752    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:05:33.262814 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:33.262783    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:05:34.282543 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:34.282504    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" event={"ID":"991fccf4-8422-4474-97ec-a0cb5b407788","Type":"ContainerStarted","Data":"f73774ea41d57606d564e9ae768a1dd45aba4bc22eb1e92fd15fc1fa4987ed5b"}
Nov 02 10:05:35.143296 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:35.143262    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:35.143675 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:35.143631    4449 patch_prober.go:29] interesting pod/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Nov 02 10:05:35.143820 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:35.143698    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Nov 02 10:05:36.143707 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:36.143670    4449 patch_prober.go:29] interesting pod/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Nov 02 10:05:36.144118 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:36.143727    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Nov 02 10:05:36.547121 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:05:36.547086    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:05:37.143871 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:37.143829    4449 patch_prober.go:29] interesting pod/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Nov 02 10:05:37.144441 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:37.143892    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Nov 02 10:05:38.143047 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:38.143002    4449 patch_prober.go:29] interesting pod/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Nov 02 10:05:38.143287 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:38.143064    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Nov 02 10:05:39.143250 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:39.143185    4449 patch_prober.go:29] interesting pod/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body=
Nov 02 10:05:39.143662 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:39.143272    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused"
Nov 02 10:05:40.148091 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:40.148062    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:40.148569 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:40.148546    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:40.149320 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:40.149304    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:51.141660 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:51.141619    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"]
Nov 02 10:05:51.142340 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:51.141883    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy" containerID="cri-o://f73774ea41d57606d564e9ae768a1dd45aba4bc22eb1e92fd15fc1fa4987ed5b" gracePeriod=30
Nov 02 10:05:55.151037 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:55.150986    4449 patch_prober.go:29] interesting pod/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 10:05:55.151488 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:55.151080    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 10:05:56.435000 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.434917    4449 generic.go:356] "Generic (PLEG): container finished" podID="991fccf4-8422-4474-97ec-a0cb5b407788" containerID="f73774ea41d57606d564e9ae768a1dd45aba4bc22eb1e92fd15fc1fa4987ed5b" exitCode=0
Nov 02 10:05:56.435422 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.435026    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" event={"ID":"991fccf4-8422-4474-97ec-a0cb5b407788","Type":"ContainerDied","Data":"f73774ea41d57606d564e9ae768a1dd45aba4bc22eb1e92fd15fc1fa4987ed5b"}
Nov 02 10:05:56.435422 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.435071    4449 scope.go:122] "RemoveContainer" containerID="16e087329a092cb4110b178b172a1f1a0320243c85985995cb1d9d0a3e81c2d9"
Nov 02 10:05:56.730822 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.730679    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.841107    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-envoy\") pod \"991fccf4-8422-4474-97ec-a0cb5b407788\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") "
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.841174    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-socket\") pod \"991fccf4-8422-4474-97ec-a0cb5b407788\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") "
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.841236    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-kube-api-access-mfktj\" (UniqueName: \"kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-kube-api-access-mfktj\") pod \"991fccf4-8422-4474-97ec-a0cb5b407788\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") "
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.841296    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-data\") pod \"991fccf4-8422-4474-97ec-a0cb5b407788\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") "
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.841377    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-istio-token\" (UniqueName: \"kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-istio-token\") pod \"991fccf4-8422-4474-97ec-a0cb5b407788\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") "
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.841485    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-certs\") pod \"991fccf4-8422-4474-97ec-a0cb5b407788\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") "
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.841544    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/991fccf4-8422-4474-97ec-a0cb5b407788-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991fccf4-8422-4474-97ec-a0cb5b407788-istio-podinfo\") pod \"991fccf4-8422-4474-97ec-a0cb5b407788\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") "
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.841606    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-credential-socket\") pod \"991fccf4-8422-4474-97ec-a0cb5b407788\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") "
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.841607    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-socket" pod "991fccf4-8422-4474-97ec-a0cb5b407788" (UID: "991fccf4-8422-4474-97ec-a0cb5b407788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.841662    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/991fccf4-8422-4474-97ec-a0cb5b407788-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991fccf4-8422-4474-97ec-a0cb5b407788-istiod-ca-cert\") pod \"991fccf4-8422-4474-97ec-a0cb5b407788\" (UID: \"991fccf4-8422-4474-97ec-a0cb5b407788\") "
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.842165    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.842696    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991fccf4-8422-4474-97ec-a0cb5b407788-istiod-ca-cert" pod "991fccf4-8422-4474-97ec-a0cb5b407788" (UID: "991fccf4-8422-4474-97ec-a0cb5b407788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.842998    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-certs" pod "991fccf4-8422-4474-97ec-a0cb5b407788" (UID: "991fccf4-8422-4474-97ec-a0cb5b407788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.843256    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-credential-socket" pod "991fccf4-8422-4474-97ec-a0cb5b407788" (UID: "991fccf4-8422-4474-97ec-a0cb5b407788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:56.844347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.843558    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-data" pod "991fccf4-8422-4474-97ec-a0cb5b407788" (UID: "991fccf4-8422-4474-97ec-a0cb5b407788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:56.846112 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.845540    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-istio-token" pod "991fccf4-8422-4474-97ec-a0cb5b407788" (UID: "991fccf4-8422-4474-97ec-a0cb5b407788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:56.846371 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.846347    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-kube-api-access-mfktj" pod "991fccf4-8422-4474-97ec-a0cb5b407788" (UID: "991fccf4-8422-4474-97ec-a0cb5b407788"). InnerVolumeSpecName "kube-api-access-mfktj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:56.846946 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.846923    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/991fccf4-8422-4474-97ec-a0cb5b407788-istio-podinfo" pod "991fccf4-8422-4474-97ec-a0cb5b407788" (UID: "991fccf4-8422-4474-97ec-a0cb5b407788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:05:56.847272 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.847208    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-envoy" pod "991fccf4-8422-4474-97ec-a0cb5b407788" (UID: "991fccf4-8422-4474-97ec-a0cb5b407788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:56.943224 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.943160    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:56.943410 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.943237    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfktj\" (UniqueName: \"kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-kube-api-access-mfktj\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:56.943410 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.943258    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:56.943410 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.943273    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991fccf4-8422-4474-97ec-a0cb5b407788-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:56.943410 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.943288    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:56.943410 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.943303    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991fccf4-8422-4474-97ec-a0cb5b407788-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:56.943410 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.943317    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991fccf4-8422-4474-97ec-a0cb5b407788-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:56.943410 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:56.943331    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991fccf4-8422-4474-97ec-a0cb5b407788-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:57.442109 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:57.442077    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"
Nov 02 10:05:57.442619 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:57.442118    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5" event={"ID":"991fccf4-8422-4474-97ec-a0cb5b407788","Type":"ContainerDied","Data":"b0875fbd5cca0575980e49d56079dead953dd5b83d6260057c52e74dbc2be546"}
Nov 02 10:05:57.442619 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:57.442192    4449 scope.go:122] "RemoveContainer" containerID="f73774ea41d57606d564e9ae768a1dd45aba4bc22eb1e92fd15fc1fa4987ed5b"
Nov 02 10:05:57.488294 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:57.476576    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"]
Nov 02 10:05:57.494518 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:57.494483    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b9ae0feb-openshift-default-558dd897b5-58kf5"]
Nov 02 10:05:59.205991 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:05:59.205951    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" path="/var/lib/kubelet/pods/991fccf4-8422-4474-97ec-a0cb5b407788/volumes"
Nov 02 10:06:06.547781 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:06:06.547692    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:06:07.025511 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:06:07.025424    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"]
Nov 02 10:06:07.028021 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:06:07.027999    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30607e68-7tmmg"]
Nov 02 10:06:07.206198 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:06:07.206161    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d" path="/var/lib/kubelet/pods/8d13c536-fd9e-4ed4-9adb-cdd0a5d1fd3d/volumes"
Nov 02 10:06:20.030591 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:06:20.030557    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"]
Nov 02 10:06:20.031014 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:06:20.030609    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad9d226-m9qbt"]
Nov 02 10:06:21.205874 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:06:21.205836    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57" path="/var/lib/kubelet/pods/9ff3cbe7-38dd-4f32-a0ad-8ca2ac7a8c57/volumes"
Nov 02 10:06:26.036954 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:06:26.036918    4449 scope.go:122] "RemoveContainer" containerID="cd50c8e658313dee1f472e885ccb3b4358f82bfd1534b9229be1eb2e2c9bd9d0"
Nov 02 10:06:36.547082 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:06:36.547039    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:07:03.083909 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.083867    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"]
Nov 02 10:07:03.084485 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.084105    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch" podUID="d1af78de-9573-422d-8563-4a32acc6e833" containerName="discovery" containerID="cri-o://e83bdb7b39407e4b99587cc4e2cdf125e26e9dd49f23a0542ea1c776a6c50ba3" gracePeriod=30
Nov 02 10:07:03.342903 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.342835    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 10:07:03.474730 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.474699    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-kube-api-access-8mtzh\" (UniqueName: \"kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-kube-api-access-8mtzh\") pod \"d1af78de-9573-422d-8563-4a32acc6e833\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") "
Nov 02 10:07:03.474931 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.474753    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-istio-token\" (UniqueName: \"kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-istio-token\") pod \"d1af78de-9573-422d-8563-4a32acc6e833\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") "
Nov 02 10:07:03.474931 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.474797    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1af78de-9573-422d-8563-4a32acc6e833-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af78de-9573-422d-8563-4a32acc6e833-local-certs\") pod \"d1af78de-9573-422d-8563-4a32acc6e833\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") "
Nov 02 10:07:03.474931 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.474860    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-kubeconfig\") pod \"d1af78de-9573-422d-8563-4a32acc6e833\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") "
Nov 02 10:07:03.474931 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.474921    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-dns-cert\") pod \"d1af78de-9573-422d-8563-4a32acc6e833\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") "
Nov 02 10:07:03.475174 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.475058    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-ca-configmap\") pod \"d1af78de-9573-422d-8563-4a32acc6e833\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") "
Nov 02 10:07:03.475174 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.475115    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-cacerts\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-cacerts\") pod \"d1af78de-9573-422d-8563-4a32acc6e833\" (UID: \"d1af78de-9573-422d-8563-4a32acc6e833\") "
Nov 02 10:07:03.475602 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.475449    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-ca-configmap" pod "d1af78de-9573-422d-8563-4a32acc6e833" (UID: "d1af78de-9573-422d-8563-4a32acc6e833"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:07:03.475827 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.475806    4449 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-ca-configmap\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:03.477431 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.477401    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-istio-token" pod "d1af78de-9573-422d-8563-4a32acc6e833" (UID: "d1af78de-9573-422d-8563-4a32acc6e833"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:07:03.477684 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.477663    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-dns-cert" pod "d1af78de-9573-422d-8563-4a32acc6e833" (UID: "d1af78de-9573-422d-8563-4a32acc6e833"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:07:03.477804 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.477708    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-kube-api-access-8mtzh" pod "d1af78de-9573-422d-8563-4a32acc6e833" (UID: "d1af78de-9573-422d-8563-4a32acc6e833"). InnerVolumeSpecName "kube-api-access-8mtzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:07:03.477872 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.477828    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-kubeconfig" pod "d1af78de-9573-422d-8563-4a32acc6e833" (UID: "d1af78de-9573-422d-8563-4a32acc6e833"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:07:03.477872 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.477843    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1af78de-9573-422d-8563-4a32acc6e833-local-certs" pod "d1af78de-9573-422d-8563-4a32acc6e833" (UID: "d1af78de-9573-422d-8563-4a32acc6e833"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:07:03.478048 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.478031    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-cacerts" pod "d1af78de-9573-422d-8563-4a32acc6e833" (UID: "d1af78de-9573-422d-8563-4a32acc6e833"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:07:03.576599 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.576565    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mtzh\" (UniqueName: \"kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-kube-api-access-8mtzh\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:03.576599 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.576597    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1af78de-9573-422d-8563-4a32acc6e833-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:03.576599 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.576607    4449 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1af78de-9573-422d-8563-4a32acc6e833-local-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:03.576864 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.576617    4449 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-kubeconfig\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:03.576864 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.576625    4449 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-istio-csr-dns-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:03.576864 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.576634    4449 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d1af78de-9573-422d-8563-4a32acc6e833-cacerts\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:03.880257 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.880198    4449 generic.go:356] "Generic (PLEG): container finished" podID="d1af78de-9573-422d-8563-4a32acc6e833" containerID="e83bdb7b39407e4b99587cc4e2cdf125e26e9dd49f23a0542ea1c776a6c50ba3" exitCode=0
Nov 02 10:07:03.880438 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.880270    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch" event={"ID":"d1af78de-9573-422d-8563-4a32acc6e833","Type":"ContainerDied","Data":"e83bdb7b39407e4b99587cc4e2cdf125e26e9dd49f23a0542ea1c776a6c50ba3"}
Nov 02 10:07:03.880438 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.880314    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch" event={"ID":"d1af78de-9573-422d-8563-4a32acc6e833","Type":"ContainerDied","Data":"23d8c56256edc2db2b0a8a33381fbf1965a9200d411802dd3e7408be695e5bf4"}
Nov 02 10:07:03.880438 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.880343    4449 scope.go:122] "RemoveContainer" containerID="e83bdb7b39407e4b99587cc4e2cdf125e26e9dd49f23a0542ea1c776a6c50ba3"
Nov 02 10:07:03.880438 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.880273    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"
Nov 02 10:07:03.890898 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.890878    4449 scope.go:122] "RemoveContainer" containerID="e83bdb7b39407e4b99587cc4e2cdf125e26e9dd49f23a0542ea1c776a6c50ba3"
Nov 02 10:07:03.891179 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:07:03.891158    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83bdb7b39407e4b99587cc4e2cdf125e26e9dd49f23a0542ea1c776a6c50ba3\": container with ID starting with e83bdb7b39407e4b99587cc4e2cdf125e26e9dd49f23a0542ea1c776a6c50ba3 not found: ID does not exist" containerID="e83bdb7b39407e4b99587cc4e2cdf125e26e9dd49f23a0542ea1c776a6c50ba3"
Nov 02 10:07:03.891281 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.891192    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83bdb7b39407e4b99587cc4e2cdf125e26e9dd49f23a0542ea1c776a6c50ba3"} err="failed to get container status \"e83bdb7b39407e4b99587cc4e2cdf125e26e9dd49f23a0542ea1c776a6c50ba3\": rpc error: code = NotFound desc = could not find container \"e83bdb7b39407e4b99587cc4e2cdf125e26e9dd49f23a0542ea1c776a6c50ba3\": container with ID starting with e83bdb7b39407e4b99587cc4e2cdf125e26e9dd49f23a0542ea1c776a6c50ba3 not found: ID does not exist"
Nov 02 10:07:03.910300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.910264    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"]
Nov 02 10:07:03.915636 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:03.915605    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-scwch"]
Nov 02 10:07:05.205308 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:05.205277    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1af78de-9573-422d-8563-4a32acc6e833" path="/var/lib/kubelet/pods/d1af78de-9573-422d-8563-4a32acc6e833/volumes"
Nov 02 10:07:06.551193 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:07:06.551149    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:07:17.792013 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.791969    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"]
Nov 02 10:07:17.792512 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.792443    4449 cpu_manager.go:414] "containerMap: removing container" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy"
Nov 02 10:07:17.792512 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.792455    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy"
Nov 02 10:07:17.792512 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.792467    4449 cpu_manager.go:414] "containerMap: removing container" podUID="d1af78de-9573-422d-8563-4a32acc6e833" containerName="discovery"
Nov 02 10:07:17.792512 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.792472    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1af78de-9573-422d-8563-4a32acc6e833" containerName="discovery"
Nov 02 10:07:17.792649 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.792561    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy"
Nov 02 10:07:17.792649 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.792578    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1af78de-9573-422d-8563-4a32acc6e833" containerName="discovery"
Nov 02 10:07:17.795868 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.795850    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"
Nov 02 10:07:17.802709 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.802671    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:07:17.802870 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.802721    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:07:17.802870 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.802833    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:07:17.802994 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.802872    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:07:17.806305 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.806281    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"]
Nov 02 10:07:17.859687 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.859654    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-r4nh4\" (UID: \"d1ed65ac-2201-41bd-b714-03c5f8af454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"
Nov 02 10:07:17.859898 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.859824    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7mz4\" (UniqueName: \"kubernetes.io/projected/d1ed65ac-2201-41bd-b714-03c5f8af454a-kube-api-access-d7mz4\") pod \"maas-api-revoke-sub-f03629db-r4nh4\" (UID: \"d1ed65ac-2201-41bd-b714-03c5f8af454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"
Nov 02 10:07:17.859986 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.859926    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-r4nh4\" (UID: \"d1ed65ac-2201-41bd-b714-03c5f8af454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"
Nov 02 10:07:17.960872 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.960822    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7mz4\" (UniqueName: \"kubernetes.io/projected/d1ed65ac-2201-41bd-b714-03c5f8af454a-kube-api-access-d7mz4\") pod \"maas-api-revoke-sub-f03629db-r4nh4\" (UID: \"d1ed65ac-2201-41bd-b714-03c5f8af454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"
Nov 02 10:07:17.961071 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.960898    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-r4nh4\" (UID: \"d1ed65ac-2201-41bd-b714-03c5f8af454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"
Nov 02 10:07:17.961071 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.960930    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-r4nh4\" (UID: \"d1ed65ac-2201-41bd-b714-03c5f8af454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"
Nov 02 10:07:17.961623 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.961600    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-r4nh4\" (UID: \"d1ed65ac-2201-41bd-b714-03c5f8af454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"
Nov 02 10:07:17.963618 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.963596    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-r4nh4\" (UID: \"d1ed65ac-2201-41bd-b714-03c5f8af454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"
Nov 02 10:07:17.973860 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:17.973827    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7mz4\" (UniqueName: \"kubernetes.io/projected/d1ed65ac-2201-41bd-b714-03c5f8af454a-kube-api-access-d7mz4\") pod \"maas-api-revoke-sub-f03629db-r4nh4\" (UID: \"d1ed65ac-2201-41bd-b714-03c5f8af454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"
Nov 02 10:07:18.112491 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:18.112392    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"
Nov 02 10:07:18.388453 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:18.388418    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"]
Nov 02 10:07:18.390375 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:07:18.390348    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ed65ac_2201_41bd_b714_03c5f8af454a.slice/crio-162c06b6f71ee64ac91381cdfe2ced01b2f9f4620c4ab157398a8bc99ef5ebcc WatchSource:0}: Error finding container 162c06b6f71ee64ac91381cdfe2ced01b2f9f4620c4ab157398a8bc99ef5ebcc: Status 404 returned error can't find the container with id 162c06b6f71ee64ac91381cdfe2ced01b2f9f4620c4ab157398a8bc99ef5ebcc
Nov 02 10:07:19.056498 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:19.056464    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4" event={"ID":"d1ed65ac-2201-41bd-b714-03c5f8af454a","Type":"ContainerStarted","Data":"0aba17c323a46ffbef5fda900fad434118ea998cda561b3866960d4fcce06ced"}
Nov 02 10:07:19.056888 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:19.056518    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4" event={"ID":"d1ed65ac-2201-41bd-b714-03c5f8af454a","Type":"ContainerStarted","Data":"162c06b6f71ee64ac91381cdfe2ced01b2f9f4620c4ab157398a8bc99ef5ebcc"}
Nov 02 10:07:19.075892 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:19.075847    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4" podStartSLOduration=2.075835021 podStartE2EDuration="2.075835021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:07:19.074604643 +0000 UTC m=+1916.012891444" watchObservedRunningTime="2026-11-02 10:07:19.075835021 +0000 UTC m=+1916.014121798"
Nov 02 10:07:20.063617 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:20.063584    4449 generic.go:356] "Generic (PLEG): container finished" podID="d1ed65ac-2201-41bd-b714-03c5f8af454a" containerID="0aba17c323a46ffbef5fda900fad434118ea998cda561b3866960d4fcce06ced" exitCode=0
Nov 02 10:07:20.064037 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:20.063638    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4" event={"ID":"d1ed65ac-2201-41bd-b714-03c5f8af454a","Type":"ContainerDied","Data":"0aba17c323a46ffbef5fda900fad434118ea998cda561b3866960d4fcce06ced"}
Nov 02 10:07:21.216144 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:21.216119    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"
Nov 02 10:07:21.295446 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:21.295404    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-xks-ca\") pod \"d1ed65ac-2201-41bd-b714-03c5f8af454a\" (UID: \"d1ed65ac-2201-41bd-b714-03c5f8af454a\") "
Nov 02 10:07:21.295627 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:21.295487    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ed65ac-2201-41bd-b714-03c5f8af454a-kube-api-access-d7mz4\" (UniqueName: \"kubernetes.io/projected/d1ed65ac-2201-41bd-b714-03c5f8af454a-kube-api-access-d7mz4\") pod \"d1ed65ac-2201-41bd-b714-03c5f8af454a\" (UID: \"d1ed65ac-2201-41bd-b714-03c5f8af454a\") "
Nov 02 10:07:21.295627 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:21.295603    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-openshift-ca\") pod \"d1ed65ac-2201-41bd-b714-03c5f8af454a\" (UID: \"d1ed65ac-2201-41bd-b714-03c5f8af454a\") "
Nov 02 10:07:21.296533 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:21.296505    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-openshift-ca" pod "d1ed65ac-2201-41bd-b714-03c5f8af454a" (UID: "d1ed65ac-2201-41bd-b714-03c5f8af454a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:07:21.297770 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:21.297745    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ed65ac-2201-41bd-b714-03c5f8af454a-kube-api-access-d7mz4" pod "d1ed65ac-2201-41bd-b714-03c5f8af454a" (UID: "d1ed65ac-2201-41bd-b714-03c5f8af454a"). InnerVolumeSpecName "kube-api-access-d7mz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:07:21.297878 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:21.297775    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-xks-ca" pod "d1ed65ac-2201-41bd-b714-03c5f8af454a" (UID: "d1ed65ac-2201-41bd-b714-03c5f8af454a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:07:21.397692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:21.397584    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:21.397692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:21.397623    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ed65ac-2201-41bd-b714-03c5f8af454a-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:21.397692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:21.397633    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7mz4\" (UniqueName: \"kubernetes.io/projected/d1ed65ac-2201-41bd-b714-03c5f8af454a-kube-api-access-d7mz4\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:22.076815 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:22.076782    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"
Nov 02 10:07:22.077006 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:22.076811    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4" event={"ID":"d1ed65ac-2201-41bd-b714-03c5f8af454a","Type":"ContainerDied","Data":"162c06b6f71ee64ac91381cdfe2ced01b2f9f4620c4ab157398a8bc99ef5ebcc"}
Nov 02 10:07:22.077006 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:22.076864    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="162c06b6f71ee64ac91381cdfe2ced01b2f9f4620c4ab157398a8bc99ef5ebcc"
Nov 02 10:07:26.063889 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:26.063857    4449 scope.go:122] "RemoveContainer" containerID="f6146e0b8200e52c0cb41f9949e7b51d5fedbf9f72109b0c66a59d7671ea6460"
Nov 02 10:07:26.073319 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:26.073298    4449 scope.go:122] "RemoveContainer" containerID="fabedce7d42576ac1964d03e069e375a8cd7c98946e863c9fe085b41e116acdb"
Nov 02 10:07:33.643403 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:33.643360    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-24rkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:07:33.643836 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:33.643431    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:07:35.121867 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.121773    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"]
Nov 02 10:07:35.122340 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.122281    4449 cpu_manager.go:414] "containerMap: removing container" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy"
Nov 02 10:07:35.122340 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.122295    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy"
Nov 02 10:07:35.122340 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.122307    4449 cpu_manager.go:414] "containerMap: removing container" podUID="d1ed65ac-2201-41bd-b714-03c5f8af454a" containerName="revoke-keys"
Nov 02 10:07:35.122340 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.122314    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ed65ac-2201-41bd-b714-03c5f8af454a" containerName="revoke-keys"
Nov 02 10:07:35.122481 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.122400    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="991fccf4-8422-4474-97ec-a0cb5b407788" containerName="istio-proxy"
Nov 02 10:07:35.122481 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.122416    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ed65ac-2201-41bd-b714-03c5f8af454a" containerName="revoke-keys"
Nov 02 10:07:35.126988 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.126972    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"
Nov 02 10:07:35.133747 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.133347    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:07:35.133747 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.133436    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:07:35.133747 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.133495    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:07:35.133747 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.133581    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:07:35.136117 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.136091    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"]
Nov 02 10:07:35.153201 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.153158    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bmspd\" (UID: \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"
Nov 02 10:07:35.153201 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.153197    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bmspd\" (UID: \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"
Nov 02 10:07:35.153482 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.153255    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwqm4\" (UniqueName: \"kubernetes.io/projected/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-kube-api-access-hwqm4\") pod \"maas-api-revoke-sub-0165d7d2-bmspd\" (UID: \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"
Nov 02 10:07:35.254258 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.254180    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bmspd\" (UID: \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"
Nov 02 10:07:35.254450 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.254270    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bmspd\" (UID: \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"
Nov 02 10:07:35.254450 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.254304    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwqm4\" (UniqueName: \"kubernetes.io/projected/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-kube-api-access-hwqm4\") pod \"maas-api-revoke-sub-0165d7d2-bmspd\" (UID: \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"
Nov 02 10:07:35.254996 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.254973    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bmspd\" (UID: \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"
Nov 02 10:07:35.256845 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.256827    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bmspd\" (UID: \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"
Nov 02 10:07:35.267192 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.267163    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwqm4\" (UniqueName: \"kubernetes.io/projected/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-kube-api-access-hwqm4\") pod \"maas-api-revoke-sub-0165d7d2-bmspd\" (UID: \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"
Nov 02 10:07:35.444237 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.444121    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"
Nov 02 10:07:35.725501 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:35.725468    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"]
Nov 02 10:07:35.726548 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:07:35.726526    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01aa5ff_d7af_4d85_817b_4dbe9f94f600.slice/crio-a8b77a6486bba169b2d4f9226c236e02c22506ec1dce70fa2c281044a437185f WatchSource:0}: Error finding container a8b77a6486bba169b2d4f9226c236e02c22506ec1dce70fa2c281044a437185f: Status 404 returned error can't find the container with id a8b77a6486bba169b2d4f9226c236e02c22506ec1dce70fa2c281044a437185f
Nov 02 10:07:36.192451 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:36.192411    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd" event={"ID":"a01aa5ff-d7af-4d85-817b-4dbe9f94f600","Type":"ContainerStarted","Data":"f93d686c6bb3104163ab5dcabaa52cf7c64521e0d7913c3dc5c135be298587fd"}
Nov 02 10:07:36.192866 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:36.192470    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd" event={"ID":"a01aa5ff-d7af-4d85-817b-4dbe9f94f600","Type":"ContainerStarted","Data":"a8b77a6486bba169b2d4f9226c236e02c22506ec1dce70fa2c281044a437185f"}
Nov 02 10:07:36.214285 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:36.214188    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd" podStartSLOduration=1.2141731390000001 podStartE2EDuration="1.214173139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:07:36.212782468 +0000 UTC m=+1933.151069265" watchObservedRunningTime="2026-11-02 10:07:36.214173139 +0000 UTC m=+1933.152459938"
Nov 02 10:07:36.547426 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:07:36.547392    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:07:37.203610 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:37.203577    4449 generic.go:356] "Generic (PLEG): container finished" podID="a01aa5ff-d7af-4d85-817b-4dbe9f94f600" containerID="f93d686c6bb3104163ab5dcabaa52cf7c64521e0d7913c3dc5c135be298587fd" exitCode=0
Nov 02 10:07:37.208534 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:37.208503    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd" event={"ID":"a01aa5ff-d7af-4d85-817b-4dbe9f94f600","Type":"ContainerDied","Data":"f93d686c6bb3104163ab5dcabaa52cf7c64521e0d7913c3dc5c135be298587fd"}
Nov 02 10:07:38.339855 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:38.339827    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"
Nov 02 10:07:38.388665 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:38.388629    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-kube-api-access-hwqm4\" (UniqueName: \"kubernetes.io/projected/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-kube-api-access-hwqm4\") pod \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\" (UID: \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\") "
Nov 02 10:07:38.388855 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:38.388700    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-xks-ca\") pod \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\" (UID: \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\") "
Nov 02 10:07:38.388855 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:38.388775    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-openshift-ca\") pod \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\" (UID: \"a01aa5ff-d7af-4d85-817b-4dbe9f94f600\") "
Nov 02 10:07:38.389176 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:38.389137    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-openshift-ca" pod "a01aa5ff-d7af-4d85-817b-4dbe9f94f600" (UID: "a01aa5ff-d7af-4d85-817b-4dbe9f94f600"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:07:38.390955 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:38.390922    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-kube-api-access-hwqm4" pod "a01aa5ff-d7af-4d85-817b-4dbe9f94f600" (UID: "a01aa5ff-d7af-4d85-817b-4dbe9f94f600"). InnerVolumeSpecName "kube-api-access-hwqm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:07:38.391073 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:38.390987    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-xks-ca" pod "a01aa5ff-d7af-4d85-817b-4dbe9f94f600" (UID: "a01aa5ff-d7af-4d85-817b-4dbe9f94f600"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:07:38.489925 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:38.489853    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:38.489925 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:38.489880    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwqm4\" (UniqueName: \"kubernetes.io/projected/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-kube-api-access-hwqm4\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:38.489925 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:38.489892    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01aa5ff-d7af-4d85-817b-4dbe9f94f600-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:39.215914 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:39.215891    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"
Nov 02 10:07:39.216070 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:39.215920    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd" event={"ID":"a01aa5ff-d7af-4d85-817b-4dbe9f94f600","Type":"ContainerDied","Data":"a8b77a6486bba169b2d4f9226c236e02c22506ec1dce70fa2c281044a437185f"}
Nov 02 10:07:39.216070 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:39.215973    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8b77a6486bba169b2d4f9226c236e02c22506ec1dce70fa2c281044a437185f"
Nov 02 10:07:45.953156 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:45.953117    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"]
Nov 02 10:07:45.953696 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:45.953612    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a01aa5ff-d7af-4d85-817b-4dbe9f94f600" containerName="revoke-keys"
Nov 02 10:07:45.953696 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:45.953626    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a01aa5ff-d7af-4d85-817b-4dbe9f94f600" containerName="revoke-keys"
Nov 02 10:07:45.953818 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:45.953714    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a01aa5ff-d7af-4d85-817b-4dbe9f94f600" containerName="revoke-keys"
Nov 02 10:07:45.956878 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:45.956860    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"
Nov 02 10:07:45.967300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:45.964466    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:07:45.967300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:45.964501    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:07:45.967300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:45.964680    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:07:45.967300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:45.964762    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:07:45.967300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:45.967274    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"]
Nov 02 10:07:46.068470 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:46.068425    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-f8h9n\" (UID: \"13c9980b-7544-48f0-abc1-e162aea11c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"
Nov 02 10:07:46.068620 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:46.068482    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-f8h9n\" (UID: \"13c9980b-7544-48f0-abc1-e162aea11c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"
Nov 02 10:07:46.068683 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:46.068622    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glzzp\" (UniqueName: \"kubernetes.io/projected/13c9980b-7544-48f0-abc1-e162aea11c26-kube-api-access-glzzp\") pod \"maas-api-revoke-sub-6b19338f-f8h9n\" (UID: \"13c9980b-7544-48f0-abc1-e162aea11c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"
Nov 02 10:07:46.169565 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:46.169527    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-f8h9n\" (UID: \"13c9980b-7544-48f0-abc1-e162aea11c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"
Nov 02 10:07:46.169754 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:46.169578    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-f8h9n\" (UID: \"13c9980b-7544-48f0-abc1-e162aea11c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"
Nov 02 10:07:46.169754 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:46.169617    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glzzp\" (UniqueName: \"kubernetes.io/projected/13c9980b-7544-48f0-abc1-e162aea11c26-kube-api-access-glzzp\") pod \"maas-api-revoke-sub-6b19338f-f8h9n\" (UID: \"13c9980b-7544-48f0-abc1-e162aea11c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"
Nov 02 10:07:46.170348 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:46.170329    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-f8h9n\" (UID: \"13c9980b-7544-48f0-abc1-e162aea11c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"
Nov 02 10:07:46.172192 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:46.172168    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-f8h9n\" (UID: \"13c9980b-7544-48f0-abc1-e162aea11c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"
Nov 02 10:07:46.183153 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:46.183123    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glzzp\" (UniqueName: \"kubernetes.io/projected/13c9980b-7544-48f0-abc1-e162aea11c26-kube-api-access-glzzp\") pod \"maas-api-revoke-sub-6b19338f-f8h9n\" (UID: \"13c9980b-7544-48f0-abc1-e162aea11c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"
Nov 02 10:07:46.275710 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:46.275628    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"
Nov 02 10:07:46.552180 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:46.552151    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"]
Nov 02 10:07:46.553945 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:07:46.553921    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c9980b_7544_48f0_abc1_e162aea11c26.slice/crio-08245a4d5e4584a0006a5bc398e992a65a927a62983d52dc070ecf1dbd3b255f WatchSource:0}: Error finding container 08245a4d5e4584a0006a5bc398e992a65a927a62983d52dc070ecf1dbd3b255f: Status 404 returned error can't find the container with id 08245a4d5e4584a0006a5bc398e992a65a927a62983d52dc070ecf1dbd3b255f
Nov 02 10:07:47.260787 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:47.260691    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n" event={"ID":"13c9980b-7544-48f0-abc1-e162aea11c26","Type":"ContainerStarted","Data":"98cc4ea2ef7c7248b8c71153f6ade6deb55ecff0b647ef98f52d39a07132e4f9"}
Nov 02 10:07:47.260787 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:47.260756    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n" event={"ID":"13c9980b-7544-48f0-abc1-e162aea11c26","Type":"ContainerStarted","Data":"08245a4d5e4584a0006a5bc398e992a65a927a62983d52dc070ecf1dbd3b255f"}
Nov 02 10:07:47.280994 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:47.280941    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n" podStartSLOduration=2.280928013 podStartE2EDuration="2.280928013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:07:47.278512783 +0000 UTC m=+1944.216799576" watchObservedRunningTime="2026-11-02 10:07:47.280928013 +0000 UTC m=+1944.219214826"
Nov 02 10:07:48.268493 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:48.268455    4449 generic.go:356] "Generic (PLEG): container finished" podID="13c9980b-7544-48f0-abc1-e162aea11c26" containerID="98cc4ea2ef7c7248b8c71153f6ade6deb55ecff0b647ef98f52d39a07132e4f9" exitCode=0
Nov 02 10:07:48.268922 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:48.268546    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n" event={"ID":"13c9980b-7544-48f0-abc1-e162aea11c26","Type":"ContainerDied","Data":"98cc4ea2ef7c7248b8c71153f6ade6deb55ecff0b647ef98f52d39a07132e4f9"}
Nov 02 10:07:49.434117 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:49.434081    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"
Nov 02 10:07:49.501585 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:49.501552    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-xks-ca\") pod \"13c9980b-7544-48f0-abc1-e162aea11c26\" (UID: \"13c9980b-7544-48f0-abc1-e162aea11c26\") "
Nov 02 10:07:49.501585 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:49.501590    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c9980b-7544-48f0-abc1-e162aea11c26-kube-api-access-glzzp\" (UniqueName: \"kubernetes.io/projected/13c9980b-7544-48f0-abc1-e162aea11c26-kube-api-access-glzzp\") pod \"13c9980b-7544-48f0-abc1-e162aea11c26\" (UID: \"13c9980b-7544-48f0-abc1-e162aea11c26\") "
Nov 02 10:07:49.501801 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:49.501753    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-openshift-ca\") pod \"13c9980b-7544-48f0-abc1-e162aea11c26\" (UID: \"13c9980b-7544-48f0-abc1-e162aea11c26\") "
Nov 02 10:07:49.502088 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:49.502062    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-openshift-ca" pod "13c9980b-7544-48f0-abc1-e162aea11c26" (UID: "13c9980b-7544-48f0-abc1-e162aea11c26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:07:49.503810 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:49.503780    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-xks-ca" pod "13c9980b-7544-48f0-abc1-e162aea11c26" (UID: "13c9980b-7544-48f0-abc1-e162aea11c26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:07:49.503937 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:49.503921    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c9980b-7544-48f0-abc1-e162aea11c26-kube-api-access-glzzp" pod "13c9980b-7544-48f0-abc1-e162aea11c26" (UID: "13c9980b-7544-48f0-abc1-e162aea11c26"). InnerVolumeSpecName "kube-api-access-glzzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:07:49.603025 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:49.602985    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:49.603025 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:49.603021    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c9980b-7544-48f0-abc1-e162aea11c26-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:49.603276 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:49.603034    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glzzp\" (UniqueName: \"kubernetes.io/projected/13c9980b-7544-48f0-abc1-e162aea11c26-kube-api-access-glzzp\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:07:50.282601 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:50.282574    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"
Nov 02 10:07:50.282756 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:50.282579    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n" event={"ID":"13c9980b-7544-48f0-abc1-e162aea11c26","Type":"ContainerDied","Data":"08245a4d5e4584a0006a5bc398e992a65a927a62983d52dc070ecf1dbd3b255f"}
Nov 02 10:07:50.282756 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:50.282704    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08245a4d5e4584a0006a5bc398e992a65a927a62983d52dc070ecf1dbd3b255f"
Nov 02 10:07:58.636672 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.636586    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"]
Nov 02 10:07:58.637323 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.637300    4449 cpu_manager.go:414] "containerMap: removing container" podUID="13c9980b-7544-48f0-abc1-e162aea11c26" containerName="revoke-keys"
Nov 02 10:07:58.637425 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.637325    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c9980b-7544-48f0-abc1-e162aea11c26" containerName="revoke-keys"
Nov 02 10:07:58.637489 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.637473    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c9980b-7544-48f0-abc1-e162aea11c26" containerName="revoke-keys"
Nov 02 10:07:58.641156 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.641131    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"]
Nov 02 10:07:58.641309 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.641248    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.646604 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.646562    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4a8104f7-kserve-self-signed-certs\""
Nov 02 10:07:58.736376 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.736340    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:07:58.736693 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.736644    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:07:58.831244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.830785    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-tmp-dir\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.831244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.830853    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4bql\" (UniqueName: \"kubernetes.io/projected/1841b104-c290-463e-aea0-4b81ce9398b2-kube-api-access-m4bql\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.831244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.830898    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-model-cache\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.831244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.830970    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1841b104-c290-463e-aea0-4b81ce9398b2-tls-certs\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.831244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.830999    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-home\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.831244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.831031    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-dshm\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.932293 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.931821    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-tmp-dir\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.932293 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.931896    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4bql\" (UniqueName: \"kubernetes.io/projected/1841b104-c290-463e-aea0-4b81ce9398b2-kube-api-access-m4bql\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.932293 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.931948    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-model-cache\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.932293 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.932031    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1841b104-c290-463e-aea0-4b81ce9398b2-tls-certs\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.932293 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.932061    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-home\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.932293 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.932097    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-dshm\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.935501 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.934720    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-model-cache\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.935501 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.935011    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-tmp-dir\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.935501 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.935425    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-home\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.937286 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.937255    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-dshm\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.938760 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.938737    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1841b104-c290-463e-aea0-4b81ce9398b2-tls-certs\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:58.947961 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:58.947937    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4bql\" (UniqueName: \"kubernetes.io/projected/1841b104-c290-463e-aea0-4b81ce9398b2-kube-api-access-m4bql\") pod \"e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") " pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:59.035311 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:59.035023    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:07:59.532508 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:07:59.532481    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"]
Nov 02 10:07:59.536095 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:07:59.536053    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1841b104_c290_463e_aea0_4b81ce9398b2.slice/crio-4af206dd5ce0259e05daafdfb2cffdce7532125ebc48a9440bfbbc0504f09dcb WatchSource:0}: Error finding container 4af206dd5ce0259e05daafdfb2cffdce7532125ebc48a9440bfbbc0504f09dcb: Status 404 returned error can't find the container with id 4af206dd5ce0259e05daafdfb2cffdce7532125ebc48a9440bfbbc0504f09dcb
Nov 02 10:08:00.351194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:00.351160    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh" event={"ID":"1841b104-c290-463e-aea0-4b81ce9398b2","Type":"ContainerStarted","Data":"bcfa43581b7437eed89e524aceba4be5d03914078760f2f1a9abb5a089361b30"}
Nov 02 10:08:00.351194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:00.351200    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh" event={"ID":"1841b104-c290-463e-aea0-4b81ce9398b2","Type":"ContainerStarted","Data":"4af206dd5ce0259e05daafdfb2cffdce7532125ebc48a9440bfbbc0504f09dcb"}
Nov 02 10:08:00.370862 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:00.370815    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh" podStartSLOduration=2.370802039 podStartE2EDuration="2.370802039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:08:00.369378193 +0000 UTC m=+1957.307664969" watchObservedRunningTime="2026-11-02 10:08:00.370802039 +0000 UTC m=+1957.309088854"
Nov 02 10:08:01.035547 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:01.035508    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:08:06.553647 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:08:06.553613    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:08:09.035947 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:09.035907    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:08:09.049122 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:09.049096    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:08:09.447486 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:09.447403    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:08:22.025043 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:22.025005    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"]
Nov 02 10:08:22.027929 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:22.027897    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1a4f6f-stbts"]
Nov 02 10:08:23.206048 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:23.206014    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea54a6f-c7ec-4a58-8d50-205640bf1e31" path="/var/lib/kubelet/pods/fea54a6f-c7ec-4a58-8d50-205640bf1e31/volumes"
Nov 02 10:08:33.028878 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:33.028843    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"]
Nov 02 10:08:33.041034 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:33.041001    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02b017c-6mnhg"]
Nov 02 10:08:33.205042 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:33.205004    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a34894f0-d536-4d23-89af-b77ca263154a" path="/var/lib/kubelet/pods/a34894f0-d536-4d23-89af-b77ca263154a/volumes"
Nov 02 10:08:36.546927 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:08:36.546885    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:08:47.026992 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:47.026953    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"]
Nov 02 10:08:47.030779 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:47.030748    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de8b560-m6q54"]
Nov 02 10:08:47.205092 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:47.205058    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99c0175-8978-42bd-aebe-e59afa74d216" path="/var/lib/kubelet/pods/f99c0175-8978-42bd-aebe-e59afa74d216/volumes"
Nov 02 10:08:50.072525 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.072486    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"]
Nov 02 10:08:50.076491 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.076470    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"
Nov 02 10:08:50.083078 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.083032    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:08:50.083078 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.083095    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:08:50.083599 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.083559    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:08:50.084127 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.083656    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:08:50.087295 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.087271    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"]
Nov 02 10:08:50.196430 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.196388    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-392f38c5-w9lff\" (UID: \"c9580d75-ff52-4612-b5e7-da7814cecb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"
Nov 02 10:08:50.196430 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.196435    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-392f38c5-w9lff\" (UID: \"c9580d75-ff52-4612-b5e7-da7814cecb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"
Nov 02 10:08:50.196702 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.196557    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn594\" (UniqueName: \"kubernetes.io/projected/c9580d75-ff52-4612-b5e7-da7814cecb34-kube-api-access-hn594\") pod \"maas-api-revoke-sub-392f38c5-w9lff\" (UID: \"c9580d75-ff52-4612-b5e7-da7814cecb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"
Nov 02 10:08:50.297233 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.297185    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-392f38c5-w9lff\" (UID: \"c9580d75-ff52-4612-b5e7-da7814cecb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"
Nov 02 10:08:50.297450 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.297265    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-392f38c5-w9lff\" (UID: \"c9580d75-ff52-4612-b5e7-da7814cecb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"
Nov 02 10:08:50.297450 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.297301    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn594\" (UniqueName: \"kubernetes.io/projected/c9580d75-ff52-4612-b5e7-da7814cecb34-kube-api-access-hn594\") pod \"maas-api-revoke-sub-392f38c5-w9lff\" (UID: \"c9580d75-ff52-4612-b5e7-da7814cecb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"
Nov 02 10:08:50.297874 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.297844    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-392f38c5-w9lff\" (UID: \"c9580d75-ff52-4612-b5e7-da7814cecb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"
Nov 02 10:08:50.299698 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.299672    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-392f38c5-w9lff\" (UID: \"c9580d75-ff52-4612-b5e7-da7814cecb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"
Nov 02 10:08:50.310844 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.310814    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn594\" (UniqueName: \"kubernetes.io/projected/c9580d75-ff52-4612-b5e7-da7814cecb34-kube-api-access-hn594\") pod \"maas-api-revoke-sub-392f38c5-w9lff\" (UID: \"c9580d75-ff52-4612-b5e7-da7814cecb34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"
Nov 02 10:08:50.391437 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.391349    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"
Nov 02 10:08:50.670135 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.670110    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"]
Nov 02 10:08:50.672018 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:08:50.671988    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9580d75_ff52_4612_b5e7_da7814cecb34.slice/crio-877346bfa00d8b8c1bfbc8e29d8227f7525941f9c80a798f411e7bd4066fdf2e WatchSource:0}: Error finding container 877346bfa00d8b8c1bfbc8e29d8227f7525941f9c80a798f411e7bd4066fdf2e: Status 404 returned error can't find the container with id 877346bfa00d8b8c1bfbc8e29d8227f7525941f9c80a798f411e7bd4066fdf2e
Nov 02 10:08:50.690164 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:50.690127    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff" event={"ID":"c9580d75-ff52-4612-b5e7-da7814cecb34","Type":"ContainerStarted","Data":"877346bfa00d8b8c1bfbc8e29d8227f7525941f9c80a798f411e7bd4066fdf2e"}
Nov 02 10:08:51.698689 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:51.698647    4449 generic.go:356] "Generic (PLEG): container finished" podID="c9580d75-ff52-4612-b5e7-da7814cecb34" containerID="a6fbf15a35e9070b766964ffdb5cfbe2f2e311b2bdc1189a78a42fdb90a13fec" exitCode=0
Nov 02 10:08:51.699163 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:51.698726    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff" event={"ID":"c9580d75-ff52-4612-b5e7-da7814cecb34","Type":"ContainerDied","Data":"a6fbf15a35e9070b766964ffdb5cfbe2f2e311b2bdc1189a78a42fdb90a13fec"}
Nov 02 10:08:52.839877 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:52.839846    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"
Nov 02 10:08:52.926947 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:52.926906    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-openshift-ca\") pod \"c9580d75-ff52-4612-b5e7-da7814cecb34\" (UID: \"c9580d75-ff52-4612-b5e7-da7814cecb34\") "
Nov 02 10:08:52.927158 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:52.926995    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-xks-ca\") pod \"c9580d75-ff52-4612-b5e7-da7814cecb34\" (UID: \"c9580d75-ff52-4612-b5e7-da7814cecb34\") "
Nov 02 10:08:52.927158 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:52.927060    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9580d75-ff52-4612-b5e7-da7814cecb34-kube-api-access-hn594\" (UniqueName: \"kubernetes.io/projected/c9580d75-ff52-4612-b5e7-da7814cecb34-kube-api-access-hn594\") pod \"c9580d75-ff52-4612-b5e7-da7814cecb34\" (UID: \"c9580d75-ff52-4612-b5e7-da7814cecb34\") "
Nov 02 10:08:52.927331 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:52.927305    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-openshift-ca" pod "c9580d75-ff52-4612-b5e7-da7814cecb34" (UID: "c9580d75-ff52-4612-b5e7-da7814cecb34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:08:52.927451 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:52.927436    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:08:52.929372 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:52.929339    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-xks-ca" pod "c9580d75-ff52-4612-b5e7-da7814cecb34" (UID: "c9580d75-ff52-4612-b5e7-da7814cecb34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:08:52.929372 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:52.929362    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9580d75-ff52-4612-b5e7-da7814cecb34-kube-api-access-hn594" pod "c9580d75-ff52-4612-b5e7-da7814cecb34" (UID: "c9580d75-ff52-4612-b5e7-da7814cecb34"). InnerVolumeSpecName "kube-api-access-hn594". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:08:53.029020 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:53.028935    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9580d75-ff52-4612-b5e7-da7814cecb34-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:08:53.029020 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:53.028966    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn594\" (UniqueName: \"kubernetes.io/projected/c9580d75-ff52-4612-b5e7-da7814cecb34-kube-api-access-hn594\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:08:53.711820 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:53.711788    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"
Nov 02 10:08:53.711820 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:53.711806    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff" event={"ID":"c9580d75-ff52-4612-b5e7-da7814cecb34","Type":"ContainerDied","Data":"877346bfa00d8b8c1bfbc8e29d8227f7525941f9c80a798f411e7bd4066fdf2e"}
Nov 02 10:08:53.712042 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:53.711848    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="877346bfa00d8b8c1bfbc8e29d8227f7525941f9c80a798f411e7bd4066fdf2e"
Nov 02 10:08:58.026362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:58.026316    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"]
Nov 02 10:08:58.029128 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:58.029099    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1735450-kljcj"]
Nov 02 10:08:59.206234 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:08:59.206190    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0991bf33-2095-4216-bbe2-6444e1de285f" path="/var/lib/kubelet/pods/0991bf33-2095-4216-bbe2-6444e1de285f/volumes"
Nov 02 10:09:01.618833 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:01.618797    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"]
Nov 02 10:09:06.549251 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:09:06.549144    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:09:06.668868 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.665340    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"]
Nov 02 10:09:06.668868 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.666491    4449 cpu_manager.go:414] "containerMap: removing container" podUID="c9580d75-ff52-4612-b5e7-da7814cecb34" containerName="revoke-keys"
Nov 02 10:09:06.668868 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.666511    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9580d75-ff52-4612-b5e7-da7814cecb34" containerName="revoke-keys"
Nov 02 10:09:06.668868 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.666742    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9580d75-ff52-4612-b5e7-da7814cecb34" containerName="revoke-keys"
Nov 02 10:09:06.676484 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.676455    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"
Nov 02 10:09:06.679461 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.679428    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"]
Nov 02 10:09:06.683576 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.683059    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:09:06.684613 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.684589    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:09:06.708748 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.708713    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:09:06.708748 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.708716    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:09:06.823720 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.823609    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9f854b-7s8wl\" (UID: \"b84ef03b-af8f-4701-a034-744bcea14bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"
Nov 02 10:09:06.823720 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.823681    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c78rw\" (UniqueName: \"kubernetes.io/projected/b84ef03b-af8f-4701-a034-744bcea14bad-kube-api-access-c78rw\") pod \"maas-api-revoke-sub-de9f854b-7s8wl\" (UID: \"b84ef03b-af8f-4701-a034-744bcea14bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"
Nov 02 10:09:06.823993 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.823778    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9f854b-7s8wl\" (UID: \"b84ef03b-af8f-4701-a034-744bcea14bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"
Nov 02 10:09:06.924516 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.924468    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9f854b-7s8wl\" (UID: \"b84ef03b-af8f-4701-a034-744bcea14bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"
Nov 02 10:09:06.924745 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.924529    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c78rw\" (UniqueName: \"kubernetes.io/projected/b84ef03b-af8f-4701-a034-744bcea14bad-kube-api-access-c78rw\") pod \"maas-api-revoke-sub-de9f854b-7s8wl\" (UID: \"b84ef03b-af8f-4701-a034-744bcea14bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"
Nov 02 10:09:06.924745 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.924560    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9f854b-7s8wl\" (UID: \"b84ef03b-af8f-4701-a034-744bcea14bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"
Nov 02 10:09:06.925395 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.925363    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9f854b-7s8wl\" (UID: \"b84ef03b-af8f-4701-a034-744bcea14bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"
Nov 02 10:09:06.927449 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.927417    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9f854b-7s8wl\" (UID: \"b84ef03b-af8f-4701-a034-744bcea14bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"
Nov 02 10:09:06.938001 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:06.937969    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c78rw\" (UniqueName: \"kubernetes.io/projected/b84ef03b-af8f-4701-a034-744bcea14bad-kube-api-access-c78rw\") pod \"maas-api-revoke-sub-de9f854b-7s8wl\" (UID: \"b84ef03b-af8f-4701-a034-744bcea14bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"
Nov 02 10:09:07.000788 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:07.000745    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"
Nov 02 10:09:07.294617 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:07.294586    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"]
Nov 02 10:09:07.297776 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:09:07.297728    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84ef03b_af8f_4701_a034_744bcea14bad.slice/crio-2387064557a99f78729c9dc912a9d093bf219410fa5dc5fe2531c13bf4d8b9f5 WatchSource:0}: Error finding container 2387064557a99f78729c9dc912a9d093bf219410fa5dc5fe2531c13bf4d8b9f5: Status 404 returned error can't find the container with id 2387064557a99f78729c9dc912a9d093bf219410fa5dc5fe2531c13bf4d8b9f5
Nov 02 10:09:07.840436 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:07.840333    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl" event={"ID":"b84ef03b-af8f-4701-a034-744bcea14bad","Type":"ContainerStarted","Data":"83d583919c2db78649d46c94f5fdb668a6308c740dcdd0bba6eaf670ac08b93a"}
Nov 02 10:09:07.840436 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:07.840396    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl" event={"ID":"b84ef03b-af8f-4701-a034-744bcea14bad","Type":"ContainerStarted","Data":"2387064557a99f78729c9dc912a9d093bf219410fa5dc5fe2531c13bf4d8b9f5"}
Nov 02 10:09:07.879245 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:07.878893    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl" podStartSLOduration=1.878871615 podStartE2EDuration="1.878871615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:09:07.858543705 +0000 UTC m=+2024.796830507" watchObservedRunningTime="2026-11-02 10:09:07.878871615 +0000 UTC m=+2024.817158391"
Nov 02 10:09:08.849805 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:08.849769    4449 generic.go:356] "Generic (PLEG): container finished" podID="b84ef03b-af8f-4701-a034-744bcea14bad" containerID="83d583919c2db78649d46c94f5fdb668a6308c740dcdd0bba6eaf670ac08b93a" exitCode=0
Nov 02 10:09:08.850573 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:08.850542    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl" event={"ID":"b84ef03b-af8f-4701-a034-744bcea14bad","Type":"ContainerDied","Data":"83d583919c2db78649d46c94f5fdb668a6308c740dcdd0bba6eaf670ac08b93a"}
Nov 02 10:09:10.002085 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.002059    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"
Nov 02 10:09:10.027028 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.026993    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"]
Nov 02 10:09:10.030314 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.030289    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-558229dc-qjwnj"]
Nov 02 10:09:10.181280 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.181164    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-openshift-ca\") pod \"b84ef03b-af8f-4701-a034-744bcea14bad\" (UID: \"b84ef03b-af8f-4701-a034-744bcea14bad\") "
Nov 02 10:09:10.181280 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.181238    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-xks-ca\") pod \"b84ef03b-af8f-4701-a034-744bcea14bad\" (UID: \"b84ef03b-af8f-4701-a034-744bcea14bad\") "
Nov 02 10:09:10.181518 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.181286    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84ef03b-af8f-4701-a034-744bcea14bad-kube-api-access-c78rw\" (UniqueName: \"kubernetes.io/projected/b84ef03b-af8f-4701-a034-744bcea14bad-kube-api-access-c78rw\") pod \"b84ef03b-af8f-4701-a034-744bcea14bad\" (UID: \"b84ef03b-af8f-4701-a034-744bcea14bad\") "
Nov 02 10:09:10.181620 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.181593    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-openshift-ca" pod "b84ef03b-af8f-4701-a034-744bcea14bad" (UID: "b84ef03b-af8f-4701-a034-744bcea14bad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:09:10.183698 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.183670    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84ef03b-af8f-4701-a034-744bcea14bad-kube-api-access-c78rw" pod "b84ef03b-af8f-4701-a034-744bcea14bad" (UID: "b84ef03b-af8f-4701-a034-744bcea14bad"). InnerVolumeSpecName "kube-api-access-c78rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:09:10.183891 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.183874    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-xks-ca" pod "b84ef03b-af8f-4701-a034-744bcea14bad" (UID: "b84ef03b-af8f-4701-a034-744bcea14bad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:09:10.282732 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.282683    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c78rw\" (UniqueName: \"kubernetes.io/projected/b84ef03b-af8f-4701-a034-744bcea14bad-kube-api-access-c78rw\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:09:10.282732 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.282733    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:09:10.282954 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.282749    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b84ef03b-af8f-4701-a034-744bcea14bad-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:09:10.864511 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.864482    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"
Nov 02 10:09:10.864741 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.864515    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl" event={"ID":"b84ef03b-af8f-4701-a034-744bcea14bad","Type":"ContainerDied","Data":"2387064557a99f78729c9dc912a9d093bf219410fa5dc5fe2531c13bf4d8b9f5"}
Nov 02 10:09:10.864741 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:10.864560    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2387064557a99f78729c9dc912a9d093bf219410fa5dc5fe2531c13bf4d8b9f5"
Nov 02 10:09:11.211912 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:11.211832    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d018347-76ec-437b-9bb4-3dd973a79c71" path="/var/lib/kubelet/pods/8d018347-76ec-437b-9bb4-3dd973a79c71/volumes"
Nov 02 10:09:16.650363 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.650319    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh" podUID="1841b104-c290-463e-aea0-4b81ce9398b2" containerName="main" containerID="cri-o://bcfa43581b7437eed89e524aceba4be5d03914078760f2f1a9abb5a089361b30" gracePeriod=45
Nov 02 10:09:16.901747 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.901666    4449 generic.go:356] "Generic (PLEG): container finished" podID="1841b104-c290-463e-aea0-4b81ce9398b2" containerID="bcfa43581b7437eed89e524aceba4be5d03914078760f2f1a9abb5a089361b30" exitCode=0
Nov 02 10:09:16.901862 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.901739    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh" event={"ID":"1841b104-c290-463e-aea0-4b81ce9398b2","Type":"ContainerDied","Data":"bcfa43581b7437eed89e524aceba4be5d03914078760f2f1a9abb5a089361b30"}
Nov 02 10:09:16.901862 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.901793    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh" event={"ID":"1841b104-c290-463e-aea0-4b81ce9398b2","Type":"ContainerDied","Data":"4af206dd5ce0259e05daafdfb2cffdce7532125ebc48a9440bfbbc0504f09dcb"}
Nov 02 10:09:16.901862 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.901809    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4af206dd5ce0259e05daafdfb2cffdce7532125ebc48a9440bfbbc0504f09dcb"
Nov 02 10:09:16.916616 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.916592    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:09:16.953186 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.953145    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1841b104-c290-463e-aea0-4b81ce9398b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/1841b104-c290-463e-aea0-4b81ce9398b2-tls-certs\") pod \"1841b104-c290-463e-aea0-4b81ce9398b2\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") "
Nov 02 10:09:16.953428 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.953198    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1841b104-c290-463e-aea0-4b81ce9398b2-kube-api-access-m4bql\" (UniqueName: \"kubernetes.io/projected/1841b104-c290-463e-aea0-4b81ce9398b2-kube-api-access-m4bql\") pod \"1841b104-c290-463e-aea0-4b81ce9398b2\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") "
Nov 02 10:09:16.953428 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.953241    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-model-cache\") pod \"1841b104-c290-463e-aea0-4b81ce9398b2\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") "
Nov 02 10:09:16.953428 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.953315    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-home\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-home\") pod \"1841b104-c290-463e-aea0-4b81ce9398b2\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") "
Nov 02 10:09:16.953428 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.953342    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-dshm\") pod \"1841b104-c290-463e-aea0-4b81ce9398b2\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") "
Nov 02 10:09:16.953428 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.953396    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-tmp-dir\") pod \"1841b104-c290-463e-aea0-4b81ce9398b2\" (UID: \"1841b104-c290-463e-aea0-4b81ce9398b2\") "
Nov 02 10:09:16.953734 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.953633    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-model-cache" pod "1841b104-c290-463e-aea0-4b81ce9398b2" (UID: "1841b104-c290-463e-aea0-4b81ce9398b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:09:16.953734 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.953654    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-home" pod "1841b104-c290-463e-aea0-4b81ce9398b2" (UID: "1841b104-c290-463e-aea0-4b81ce9398b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:09:16.953852 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.953818    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:09:16.953852 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.953839    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:09:16.953852 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.953841    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-tmp-dir" pod "1841b104-c290-463e-aea0-4b81ce9398b2" (UID: "1841b104-c290-463e-aea0-4b81ce9398b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:09:16.955822 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.955788    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1841b104-c290-463e-aea0-4b81ce9398b2-kube-api-access-m4bql" pod "1841b104-c290-463e-aea0-4b81ce9398b2" (UID: "1841b104-c290-463e-aea0-4b81ce9398b2"). InnerVolumeSpecName "kube-api-access-m4bql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:09:16.955822 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.955797    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1841b104-c290-463e-aea0-4b81ce9398b2-tls-certs" pod "1841b104-c290-463e-aea0-4b81ce9398b2" (UID: "1841b104-c290-463e-aea0-4b81ce9398b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:09:16.955822 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:16.955812    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-dshm" pod "1841b104-c290-463e-aea0-4b81ce9398b2" (UID: "1841b104-c290-463e-aea0-4b81ce9398b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:09:17.054636 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:17.054582    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:09:17.054636 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:17.054632    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1841b104-c290-463e-aea0-4b81ce9398b2-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:09:17.054636 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:17.054642    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4bql\" (UniqueName: \"kubernetes.io/projected/1841b104-c290-463e-aea0-4b81ce9398b2-kube-api-access-m4bql\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:09:17.054636 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:17.054650    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1841b104-c290-463e-aea0-4b81ce9398b2-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:09:17.907659 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:17.907624    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"
Nov 02 10:09:17.927169 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:17.927121    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"]
Nov 02 10:09:17.932603 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:17.932567    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-4a8104f7-kserve-c99f876-5jksh"]
Nov 02 10:09:18.901876 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.901835    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"]
Nov 02 10:09:18.902317 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.902304    4449 cpu_manager.go:414] "containerMap: removing container" podUID="b84ef03b-af8f-4701-a034-744bcea14bad" containerName="revoke-keys"
Nov 02 10:09:18.902373 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.902318    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84ef03b-af8f-4701-a034-744bcea14bad" containerName="revoke-keys"
Nov 02 10:09:18.902373 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.902328    4449 cpu_manager.go:414] "containerMap: removing container" podUID="1841b104-c290-463e-aea0-4b81ce9398b2" containerName="main"
Nov 02 10:09:18.902373 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.902333    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1841b104-c290-463e-aea0-4b81ce9398b2" containerName="main"
Nov 02 10:09:18.902476 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.902402    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84ef03b-af8f-4701-a034-744bcea14bad" containerName="revoke-keys"
Nov 02 10:09:18.902476 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.902412    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="1841b104-c290-463e-aea0-4b81ce9398b2" containerName="main"
Nov 02 10:09:18.905843 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.905824    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:18.924693 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.924666    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-dd40e2-openshift-default-dockercfg-cbc8j\""
Nov 02 10:09:18.933270 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.933240    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"]
Nov 02 10:09:18.973553 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.973518    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istiod-ca-cert\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:18.973734 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.973562    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-token\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:18.973734 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.973625    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-envoy\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:18.973815 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.973740    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-credential-socket\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:18.973815 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.973778    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-socket\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:18.973815 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.973800    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-podinfo\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:18.973917 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.973820    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-data\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:18.973953 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.973925    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x44kq\" (UniqueName: \"kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-kube-api-access-x44kq\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:18.973992 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:18.973953    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-certs\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.074712 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.074672    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-certs\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.074920 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.074741    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istiod-ca-cert\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.074920 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.074767    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-token\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.074920 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.074786    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-envoy\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.075111 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.074940    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-credential-socket\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.075111 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.074993    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-socket\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.075111 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.075026    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-podinfo\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.075111 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.075053    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-data\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.075385 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.075111    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x44kq\" (UniqueName: \"kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-kube-api-access-x44kq\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.075385 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.075146    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-certs\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.075511 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.075438    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-socket\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.075730 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.075704    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istiod-ca-cert\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.075948 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.075920    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-data\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.076086 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.075721    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-credential-socket\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.077719 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.077691    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-envoy\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.078285 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.078258    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-podinfo\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.088298 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.088266    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x44kq\" (UniqueName: \"kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-kube-api-access-x44kq\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.088433 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.088326    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-token\") pod \"e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") " pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.205833 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.205739    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1841b104-c290-463e-aea0-4b81ce9398b2" path="/var/lib/kubelet/pods/1841b104-c290-463e-aea0-4b81ce9398b2/volumes"
Nov 02 10:09:19.219063 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.219027    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:19.505183 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.505154    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"]
Nov 02 10:09:19.506563 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:09:19.506529    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9cd89b_23d7_4cb2_a020_35ea8c107d81.slice/crio-2054cf7c4718240be444b073c9866d18a51ada2526a0895c1a6703b5af448513 WatchSource:0}: Error finding container 2054cf7c4718240be444b073c9866d18a51ada2526a0895c1a6703b5af448513: Status 404 returned error can't find the container with id 2054cf7c4718240be444b073c9866d18a51ada2526a0895c1a6703b5af448513
Nov 02 10:09:19.508968 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.508928    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:09:19.509050 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.509022    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:09:19.509090 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.509068    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:09:19.921592 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.921558    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" event={"ID":"de9cd89b-23d7-4cb2-a020-35ea8c107d81","Type":"ContainerStarted","Data":"ad65e14cc23cfdded878b38b82973442ac17f2cf22302f00e27f83d33fe9615e"}
Nov 02 10:09:19.921805 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:19.921610    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" event={"ID":"de9cd89b-23d7-4cb2-a020-35ea8c107d81","Type":"ContainerStarted","Data":"2054cf7c4718240be444b073c9866d18a51ada2526a0895c1a6703b5af448513"}
Nov 02 10:09:20.023932 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:20.023873    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" podStartSLOduration=2.02385623 podStartE2EDuration="2.02385623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:09:19.94899291 +0000 UTC m=+2036.887279703" watchObservedRunningTime="2026-11-02 10:09:20.02385623 +0000 UTC m=+2036.962143005"
Nov 02 10:09:20.025393 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:20.025369    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"]
Nov 02 10:09:20.030386 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:20.030355    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fc0c347-28688"]
Nov 02 10:09:20.220033 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:20.219927    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:20.225500 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:20.225470    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:20.928311 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:20.928280    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:20.929434 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:20.929412    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:21.205651 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:21.205573    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cc0e764-96e1-46cf-a580-96253011d988" path="/var/lib/kubelet/pods/1cc0e764-96e1-46cf-a580-96253011d988/volumes"
Nov 02 10:09:23.491435 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.491381    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"]
Nov 02 10:09:23.496476 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.496452    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"
Nov 02 10:09:23.504163 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.504140    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-dd40e2\""
Nov 02 10:09:23.505666 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.505634    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-2983d4f2-dockercfg-p2r9t\""
Nov 02 10:09:23.509583 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.509558    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"]
Nov 02 10:09:23.513885 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.513856    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"]
Nov 02 10:09:23.518528 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.518501    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"
Nov 02 10:09:23.520659 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.520628    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"]
Nov 02 10:09:23.521034 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.521003    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e6ee34-2b8b-471f-88c9-87eb537d53ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7\" (UID: \"32e6ee34-2b8b-471f-88c9-87eb537d53ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"
Nov 02 10:09:23.522510 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.521160    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdsrs\" (UniqueName: \"kubernetes.io/projected/32e6ee34-2b8b-471f-88c9-87eb537d53ea-kube-api-access-sdsrs\") pod \"payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7\" (UID: \"32e6ee34-2b8b-471f-88c9-87eb537d53ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"
Nov 02 10:09:23.622290 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.622250    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e6ee34-2b8b-471f-88c9-87eb537d53ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7\" (UID: \"32e6ee34-2b8b-471f-88c9-87eb537d53ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"
Nov 02 10:09:23.622506 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.622326    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b1dac6-6095-4c85-ae74-f814e883d9ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl\" (UID: \"f4b1dac6-6095-4c85-ae74-f814e883d9ce\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"
Nov 02 10:09:23.622506 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.622373    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w77fk\" (UniqueName: \"kubernetes.io/projected/f4b1dac6-6095-4c85-ae74-f814e883d9ce-kube-api-access-w77fk\") pod \"payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl\" (UID: \"f4b1dac6-6095-4c85-ae74-f814e883d9ce\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"
Nov 02 10:09:23.622506 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.622461    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdsrs\" (UniqueName: \"kubernetes.io/projected/32e6ee34-2b8b-471f-88c9-87eb537d53ea-kube-api-access-sdsrs\") pod \"payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7\" (UID: \"32e6ee34-2b8b-471f-88c9-87eb537d53ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"
Nov 02 10:09:23.623041 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.623014    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e6ee34-2b8b-471f-88c9-87eb537d53ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7\" (UID: \"32e6ee34-2b8b-471f-88c9-87eb537d53ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"
Nov 02 10:09:23.635109 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.635085    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdsrs\" (UniqueName: \"kubernetes.io/projected/32e6ee34-2b8b-471f-88c9-87eb537d53ea-kube-api-access-sdsrs\") pod \"payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7\" (UID: \"32e6ee34-2b8b-471f-88c9-87eb537d53ea\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"
Nov 02 10:09:23.723244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.723193    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b1dac6-6095-4c85-ae74-f814e883d9ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl\" (UID: \"f4b1dac6-6095-4c85-ae74-f814e883d9ce\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"
Nov 02 10:09:23.723522 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.723282    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w77fk\" (UniqueName: \"kubernetes.io/projected/f4b1dac6-6095-4c85-ae74-f814e883d9ce-kube-api-access-w77fk\") pod \"payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl\" (UID: \"f4b1dac6-6095-4c85-ae74-f814e883d9ce\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"
Nov 02 10:09:23.723917 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.723894    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b1dac6-6095-4c85-ae74-f814e883d9ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl\" (UID: \"f4b1dac6-6095-4c85-ae74-f814e883d9ce\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"
Nov 02 10:09:23.735900 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.735870    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w77fk\" (UniqueName: \"kubernetes.io/projected/f4b1dac6-6095-4c85-ae74-f814e883d9ce-kube-api-access-w77fk\") pod \"payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl\" (UID: \"f4b1dac6-6095-4c85-ae74-f814e883d9ce\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"
Nov 02 10:09:23.812035 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.812001    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"
Nov 02 10:09:23.833723 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:23.833696    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"
Nov 02 10:09:24.095113 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:24.095084    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"]
Nov 02 10:09:24.096747 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:09:24.096717    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e6ee34_2b8b_471f_88c9_87eb537d53ea.slice/crio-ee6f5d727d62188589baa46dbbf3174a843eb6f62fc7bf3e4b6a1b50da1fa73a WatchSource:0}: Error finding container ee6f5d727d62188589baa46dbbf3174a843eb6f62fc7bf3e4b6a1b50da1fa73a: Status 404 returned error can't find the container with id ee6f5d727d62188589baa46dbbf3174a843eb6f62fc7bf3e4b6a1b50da1fa73a
Nov 02 10:09:24.133504 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:24.133106    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"]
Nov 02 10:09:24.954674 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:24.954636    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl" event={"ID":"f4b1dac6-6095-4c85-ae74-f814e883d9ce","Type":"ContainerStarted","Data":"e664ad9d95c4a0bbcf1407ec3bbf5de24b9a5fdd220bbfced40e46fd78b680e0"}
Nov 02 10:09:24.955155 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:24.954688    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl" event={"ID":"f4b1dac6-6095-4c85-ae74-f814e883d9ce","Type":"ContainerStarted","Data":"be28b6492d5d197dbc88e828b766e214210665c2cb5dc46655e19e4e376074e2"}
Nov 02 10:09:24.955155 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:24.954741    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"
Nov 02 10:09:24.956475 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:24.956448    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7" event={"ID":"32e6ee34-2b8b-471f-88c9-87eb537d53ea","Type":"ContainerStarted","Data":"7e8fff517f5965675eb69774307fb56c611e3def64b7942385f0a4a9971d27b3"}
Nov 02 10:09:24.956586 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:24.956499    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7" event={"ID":"32e6ee34-2b8b-471f-88c9-87eb537d53ea","Type":"ContainerStarted","Data":"ee6f5d727d62188589baa46dbbf3174a843eb6f62fc7bf3e4b6a1b50da1fa73a"}
Nov 02 10:09:24.956634 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:24.956585    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"
Nov 02 10:09:24.975113 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:24.975049    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl" podStartSLOduration=1.975033495 podStartE2EDuration="1.975033495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:09:24.9727659 +0000 UTC m=+2041.911052696" watchObservedRunningTime="2026-11-02 10:09:24.975033495 +0000 UTC m=+2041.913320270"
Nov 02 10:09:24.996131 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:24.996075    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7" podStartSLOduration=1.996063705 podStartE2EDuration="1.996063705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:09:24.993908232 +0000 UTC m=+2041.932195009" watchObservedRunningTime="2026-11-02 10:09:24.996063705 +0000 UTC m=+2041.934350480"
Nov 02 10:09:26.114397 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:26.114357    4449 scope.go:122] "RemoveContainer" containerID="73f2f008e2aec46919f7e9b9f8519795ee2cfbb40d4bcfef8704c6a4cc1f5736"
Nov 02 10:09:33.675642 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.675601    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"]
Nov 02 10:09:33.682196 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.682165    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.689634 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.689600    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"]
Nov 02 10:09:33.690480 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.690456    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-dd40e2\"/\"openshift-service-ca.crt\""
Nov 02 10:09:33.690623 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.690583    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-dd40e2\"/\"kube-root-ca.crt\""
Nov 02 10:09:33.690710 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.690690    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-dd40e2\"/\"simulator-single-main-dd40e2-kserve-self-signed-certs\""
Nov 02 10:09:33.696420 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.696395    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-dd40e2\"/\"default-dockercfg-sdgtd\""
Nov 02 10:09:33.748508 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.748468    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tmp-dir\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.748726 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.748517    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-home\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.748726 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.748596    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tls-certs\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.748856 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.748765    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-model-cache\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.748856 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.748822    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstc6\" (UniqueName: \"kubernetes.io/projected/08fb94e1-62d9-4dc6-b871-5e72800e4e84-kube-api-access-xstc6\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.748936 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.748915    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-dshm\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.849981 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.849945    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tmp-dir\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.849981 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.849993    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-home\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.850288 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.850020    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tls-certs\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.850288 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.850114    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-model-cache\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.850288 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.850161    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstc6\" (UniqueName: \"kubernetes.io/projected/08fb94e1-62d9-4dc6-b871-5e72800e4e84-kube-api-access-xstc6\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.850426 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.850284    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-dshm\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.850548 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.850522    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tmp-dir\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.850622 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.850518    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-home\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.850683 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.850649    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-model-cache\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.852691 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.852669    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-dshm\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.853098 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.853074    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tls-certs\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.863394 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.863355    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstc6\" (UniqueName: \"kubernetes.io/projected/08fb94e1-62d9-4dc6-b871-5e72800e4e84-kube-api-access-xstc6\") pod \"simulator-single-main-dd40e2-kserve-77b8557869-4wg4t\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") " pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:33.999640 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:33.999541    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:34.286184 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:34.286153    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"]
Nov 02 10:09:34.287985 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:09:34.287947    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08fb94e1_62d9_4dc6_b871_5e72800e4e84.slice/crio-9f80a483fceb7c524fbe5a824bda2c28f0b768d241237035dfa6455eaef425ee WatchSource:0}: Error finding container 9f80a483fceb7c524fbe5a824bda2c28f0b768d241237035dfa6455eaef425ee: Status 404 returned error can't find the container with id 9f80a483fceb7c524fbe5a824bda2c28f0b768d241237035dfa6455eaef425ee
Nov 02 10:09:35.035683 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:35.035638    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t" event={"ID":"08fb94e1-62d9-4dc6-b871-5e72800e4e84","Type":"ContainerStarted","Data":"ccebd4ead288c9ff8c987d9bdcf917c9f36659796952aa00039bd27a0c52a8c3"}
Nov 02 10:09:35.036182 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:35.035694    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t" event={"ID":"08fb94e1-62d9-4dc6-b871-5e72800e4e84","Type":"ContainerStarted","Data":"9f80a483fceb7c524fbe5a824bda2c28f0b768d241237035dfa6455eaef425ee"}
Nov 02 10:09:35.930250 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:35.930187    4449 patch_prober.go:29] interesting pod/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 10:09:35.930521 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:35.930291    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 10:09:35.965745 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:35.965715    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"
Nov 02 10:09:35.965915 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:35.965788    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"
Nov 02 10:09:35.984235 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:35.984158    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t" podStartSLOduration=2.984143434 podStartE2EDuration="2.984143434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:09:35.059270129 +0000 UTC m=+2051.997556906" watchObservedRunningTime="2026-11-02 10:09:35.984143434 +0000 UTC m=+2052.922430209"
Nov 02 10:09:35.999940 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:35.999906    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:36.043495 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:36.043464    4449 generic.go:356] "Generic (PLEG): container finished" podID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerID="ad65e14cc23cfdded878b38b82973442ac17f2cf22302f00e27f83d33fe9615e" exitCode=0
Nov 02 10:09:36.043909 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:36.043531    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" event={"ID":"de9cd89b-23d7-4cb2-a020-35ea8c107d81","Type":"ContainerDied","Data":"ad65e14cc23cfdded878b38b82973442ac17f2cf22302f00e27f83d33fe9615e"}
Nov 02 10:09:36.044148 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:36.044127    4449 scope.go:122] "RemoveContainer" containerID="ad65e14cc23cfdded878b38b82973442ac17f2cf22302f00e27f83d33fe9615e"
Nov 02 10:09:36.045659 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:36.045625    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:09:36.045771 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:36.045699    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:09:36.045771 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:36.045735    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:09:36.546599 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:09:36.546561    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:09:37.076725 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:37.076686    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" event={"ID":"de9cd89b-23d7-4cb2-a020-35ea8c107d81","Type":"ContainerStarted","Data":"54a36bb542298db3cdf2852fef4d5bcd18155307fb4acbb2f0176a7c00857713"}
Nov 02 10:09:37.219481 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:37.219449    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:37.219713 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:37.219690    4449 patch_prober.go:29] interesting pod/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 10:09:37.219769 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:37.219751    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 10:09:38.220128 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:38.220092    4449 patch_prober.go:29] interesting pod/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 10:09:38.220561 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:38.220147    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 10:09:39.219758 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:39.219721    4449 patch_prober.go:29] interesting pod/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 10:09:39.219946 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:39.219777    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 10:09:40.219703 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:40.219654    4449 patch_prober.go:29] interesting pod/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 10:09:40.220111 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:40.219734    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 10:09:41.220117 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:41.220084    4449 patch_prober.go:29] interesting pod/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 10:09:41.220570 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:41.220141    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 10:09:42.219611 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:42.219572    4449 patch_prober.go:29] interesting pod/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 02 10:09:42.219817 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:42.219645    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 02 10:09:43.224949 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:43.224915    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:43.225360 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:43.225166    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:43.225922 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:43.225905    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:09:44.000189 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:44.000150    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:44.013043 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:44.013009    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:09:44.134875 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:09:44.134847    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:10:04.818653 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:04.818616    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"]
Nov 02 10:10:04.822876 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:04.822854    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"
Nov 02 10:10:04.828838 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:04.828816    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:10:04.828996 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:04.828973    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:10:04.828996 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:04.828833    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:10:04.829207 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:04.829034    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:10:04.831640 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:04.831619    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"]
Nov 02 10:10:04.930760 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:04.930719    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rggsg\" (UniqueName: \"kubernetes.io/projected/a7030264-16be-48b6-ba8d-d526df941d82-kube-api-access-rggsg\") pod \"maas-api-revoke-sub-651fee84-ldqmj\" (UID: \"a7030264-16be-48b6-ba8d-d526df941d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"
Nov 02 10:10:04.930968 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:04.930773    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-651fee84-ldqmj\" (UID: \"a7030264-16be-48b6-ba8d-d526df941d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"
Nov 02 10:10:04.931046 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:04.930994    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-651fee84-ldqmj\" (UID: \"a7030264-16be-48b6-ba8d-d526df941d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"
Nov 02 10:10:05.031724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:05.031685    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-651fee84-ldqmj\" (UID: \"a7030264-16be-48b6-ba8d-d526df941d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"
Nov 02 10:10:05.031724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:05.031727    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rggsg\" (UniqueName: \"kubernetes.io/projected/a7030264-16be-48b6-ba8d-d526df941d82-kube-api-access-rggsg\") pod \"maas-api-revoke-sub-651fee84-ldqmj\" (UID: \"a7030264-16be-48b6-ba8d-d526df941d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"
Nov 02 10:10:05.031954 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:05.031751    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-651fee84-ldqmj\" (UID: \"a7030264-16be-48b6-ba8d-d526df941d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"
Nov 02 10:10:05.032434 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:05.032407    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-651fee84-ldqmj\" (UID: \"a7030264-16be-48b6-ba8d-d526df941d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"
Nov 02 10:10:05.034358 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:05.034334    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-651fee84-ldqmj\" (UID: \"a7030264-16be-48b6-ba8d-d526df941d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"
Nov 02 10:10:05.044363 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:05.044334    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rggsg\" (UniqueName: \"kubernetes.io/projected/a7030264-16be-48b6-ba8d-d526df941d82-kube-api-access-rggsg\") pod \"maas-api-revoke-sub-651fee84-ldqmj\" (UID: \"a7030264-16be-48b6-ba8d-d526df941d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"
Nov 02 10:10:05.139388 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:05.139299    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"
Nov 02 10:10:05.421048 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:05.421019    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"]
Nov 02 10:10:05.422025 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:10:05.422003    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7030264_16be_48b6_ba8d_d526df941d82.slice/crio-88b968a3b78930defbb64961d16e14fa82efb87d8ebd3d9b657945ccd4736431 WatchSource:0}: Error finding container 88b968a3b78930defbb64961d16e14fa82efb87d8ebd3d9b657945ccd4736431: Status 404 returned error can't find the container with id 88b968a3b78930defbb64961d16e14fa82efb87d8ebd3d9b657945ccd4736431
Nov 02 10:10:06.255377 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:06.255342    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj" event={"ID":"a7030264-16be-48b6-ba8d-d526df941d82","Type":"ContainerStarted","Data":"45458c6b9789df63c55a7d8dbe62b9ea5ca3c83593946a6931ae48d3d580d056"}
Nov 02 10:10:06.255787 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:06.255395    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj" event={"ID":"a7030264-16be-48b6-ba8d-d526df941d82","Type":"ContainerStarted","Data":"88b968a3b78930defbb64961d16e14fa82efb87d8ebd3d9b657945ccd4736431"}
Nov 02 10:10:06.273775 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:06.273696    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj" podStartSLOduration=2.273681691 podStartE2EDuration="2.273681691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:10:06.273490738 +0000 UTC m=+2083.211777515" watchObservedRunningTime="2026-11-02 10:10:06.273681691 +0000 UTC m=+2083.211968466"
Nov 02 10:10:06.546149 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:10:06.546117    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:10:07.262422 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:07.262392    4449 generic.go:356] "Generic (PLEG): container finished" podID="a7030264-16be-48b6-ba8d-d526df941d82" containerID="45458c6b9789df63c55a7d8dbe62b9ea5ca3c83593946a6931ae48d3d580d056" exitCode=0
Nov 02 10:10:07.262909 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:07.262470    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj" event={"ID":"a7030264-16be-48b6-ba8d-d526df941d82","Type":"ContainerDied","Data":"45458c6b9789df63c55a7d8dbe62b9ea5ca3c83593946a6931ae48d3d580d056"}
Nov 02 10:10:08.410851 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:08.410823    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"
Nov 02 10:10:08.470103 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:08.470065    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7030264-16be-48b6-ba8d-d526df941d82-kube-api-access-rggsg\" (UniqueName: \"kubernetes.io/projected/a7030264-16be-48b6-ba8d-d526df941d82-kube-api-access-rggsg\") pod \"a7030264-16be-48b6-ba8d-d526df941d82\" (UID: \"a7030264-16be-48b6-ba8d-d526df941d82\") "
Nov 02 10:10:08.470310 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:08.470181    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-openshift-ca\") pod \"a7030264-16be-48b6-ba8d-d526df941d82\" (UID: \"a7030264-16be-48b6-ba8d-d526df941d82\") "
Nov 02 10:10:08.470310 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:08.470231    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-xks-ca\") pod \"a7030264-16be-48b6-ba8d-d526df941d82\" (UID: \"a7030264-16be-48b6-ba8d-d526df941d82\") "
Nov 02 10:10:08.470649 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:08.470614    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-openshift-ca" pod "a7030264-16be-48b6-ba8d-d526df941d82" (UID: "a7030264-16be-48b6-ba8d-d526df941d82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:10:08.472475 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:08.472440    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7030264-16be-48b6-ba8d-d526df941d82-kube-api-access-rggsg" pod "a7030264-16be-48b6-ba8d-d526df941d82" (UID: "a7030264-16be-48b6-ba8d-d526df941d82"). InnerVolumeSpecName "kube-api-access-rggsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:10:08.472581 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:08.472526    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-xks-ca" pod "a7030264-16be-48b6-ba8d-d526df941d82" (UID: "a7030264-16be-48b6-ba8d-d526df941d82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:10:08.572067 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:08.572013    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rggsg\" (UniqueName: \"kubernetes.io/projected/a7030264-16be-48b6-ba8d-d526df941d82-kube-api-access-rggsg\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:08.572067 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:08.572061    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:08.572329 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:08.572074    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7030264-16be-48b6-ba8d-d526df941d82-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:09.275253 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:09.275197    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj" event={"ID":"a7030264-16be-48b6-ba8d-d526df941d82","Type":"ContainerDied","Data":"88b968a3b78930defbb64961d16e14fa82efb87d8ebd3d9b657945ccd4736431"}
Nov 02 10:10:09.275430 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:09.275273    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88b968a3b78930defbb64961d16e14fa82efb87d8ebd3d9b657945ccd4736431"
Nov 02 10:10:09.275430 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:09.275226    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"
Nov 02 10:10:20.547542 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:20.547501    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"]
Nov 02 10:10:21.145236 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:21.145189    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"]
Nov 02 10:10:21.162278 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:21.162246    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"]
Nov 02 10:10:26.145156 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:26.145108    4449 scope.go:122] "RemoveContainer" containerID="51c107d38e56ad7f29e8c76746e9a7c9878d2e4273164e8878ebc6a719b7c4cf"
Nov 02 10:10:26.145653 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:26.145511    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl" podUID="f4b1dac6-6095-4c85-ae74-f814e883d9ce" containerName="payload-processing" containerID="cri-o://e664ad9d95c4a0bbcf1407ec3bbf5de24b9a5fdd220bbfced40e46fd78b680e0" gracePeriod=25
Nov 02 10:10:26.154770 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:26.154753    4449 scope.go:122] "RemoveContainer" containerID="8dc95bb0ab2ffcafbaf528b62600174e8b17b5ad213d87e1188afe63e36b90bc"
Nov 02 10:10:26.162802 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:26.162774    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7" podUID="32e6ee34-2b8b-471f-88c9-87eb537d53ea" containerName="payload-pre-processing" containerID="cri-o://7e8fff517f5965675eb69774307fb56c611e3def64b7942385f0a4a9971d27b3" gracePeriod=25
Nov 02 10:10:26.163161 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:26.163144    4449 scope.go:122] "RemoveContainer" containerID="e28ea232e59a365eb71115096d7c317dd3474dfd9e4c0cc28d64e84a47e90dc6"
Nov 02 10:10:26.171809 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:26.171794    4449 scope.go:122] "RemoveContainer" containerID="cd53752929d7ff906b9076e56a09b723786b764723d56bfdf1c739a94d2ea178"
Nov 02 10:10:26.180520 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:26.180493    4449 scope.go:122] "RemoveContainer" containerID="9f9135e72823c10dd921c90800636c72deb47253c4f035f792a804146037d7a1"
Nov 02 10:10:35.568734 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.568646    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t" podUID="08fb94e1-62d9-4dc6-b871-5e72800e4e84" containerName="main" containerID="cri-o://ccebd4ead288c9ff8c987d9bdcf917c9f36659796952aa00039bd27a0c52a8c3" gracePeriod=45
Nov 02 10:10:35.816211 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.816183    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:10:35.859264 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.859003    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-model-cache\") pod \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") "
Nov 02 10:10:35.859264 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.859065    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tls-certs\" (UniqueName: \"kubernetes.io/secret/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tls-certs\") pod \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") "
Nov 02 10:10:35.859264 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.859180    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tmp-dir\") pod \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") "
Nov 02 10:10:35.859264 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.859211    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-dshm\") pod \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") "
Nov 02 10:10:35.859650 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.859285    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08fb94e1-62d9-4dc6-b871-5e72800e4e84-kube-api-access-xstc6\" (UniqueName: \"kubernetes.io/projected/08fb94e1-62d9-4dc6-b871-5e72800e4e84-kube-api-access-xstc6\") pod \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") "
Nov 02 10:10:35.859650 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.859332    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-home\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-home\") pod \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\" (UID: \"08fb94e1-62d9-4dc6-b871-5e72800e4e84\") "
Nov 02 10:10:35.859650 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.859432    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tmp-dir" pod "08fb94e1-62d9-4dc6-b871-5e72800e4e84" (UID: "08fb94e1-62d9-4dc6-b871-5e72800e4e84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:10:35.859650 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.859441    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-model-cache" pod "08fb94e1-62d9-4dc6-b871-5e72800e4e84" (UID: "08fb94e1-62d9-4dc6-b871-5e72800e4e84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:10:35.859884 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.859712    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:35.859884 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.859731    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:35.859884 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.859800    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-home" pod "08fb94e1-62d9-4dc6-b871-5e72800e4e84" (UID: "08fb94e1-62d9-4dc6-b871-5e72800e4e84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:10:35.861875 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.861847    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fb94e1-62d9-4dc6-b871-5e72800e4e84-kube-api-access-xstc6" pod "08fb94e1-62d9-4dc6-b871-5e72800e4e84" (UID: "08fb94e1-62d9-4dc6-b871-5e72800e4e84"). InnerVolumeSpecName "kube-api-access-xstc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:10:35.862075 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.862050    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-dshm" pod "08fb94e1-62d9-4dc6-b871-5e72800e4e84" (UID: "08fb94e1-62d9-4dc6-b871-5e72800e4e84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:10:35.862155 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.862136    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tls-certs" pod "08fb94e1-62d9-4dc6-b871-5e72800e4e84" (UID: "08fb94e1-62d9-4dc6-b871-5e72800e4e84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:10:35.960320 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.960287    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xstc6\" (UniqueName: \"kubernetes.io/projected/08fb94e1-62d9-4dc6-b871-5e72800e4e84-kube-api-access-xstc6\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:35.960320 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.960317    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:35.960320 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.960327    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08fb94e1-62d9-4dc6-b871-5e72800e4e84-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:35.960585 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.960334    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08fb94e1-62d9-4dc6-b871-5e72800e4e84-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:35.964000 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.963970    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7" podUID="32e6ee34-2b8b-471f-88c9-87eb537d53ea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:10:35.964128 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:35.963972    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl" podUID="f4b1dac6-6095-4c85-ae74-f814e883d9ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:10:36.436872 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:36.436828    4449 generic.go:356] "Generic (PLEG): container finished" podID="08fb94e1-62d9-4dc6-b871-5e72800e4e84" containerID="ccebd4ead288c9ff8c987d9bdcf917c9f36659796952aa00039bd27a0c52a8c3" exitCode=0
Nov 02 10:10:36.437066 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:36.436878    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"
Nov 02 10:10:36.437066 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:36.436895    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t" event={"ID":"08fb94e1-62d9-4dc6-b871-5e72800e4e84","Type":"ContainerDied","Data":"ccebd4ead288c9ff8c987d9bdcf917c9f36659796952aa00039bd27a0c52a8c3"}
Nov 02 10:10:36.437066 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:36.436943    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t" event={"ID":"08fb94e1-62d9-4dc6-b871-5e72800e4e84","Type":"ContainerDied","Data":"9f80a483fceb7c524fbe5a824bda2c28f0b768d241237035dfa6455eaef425ee"}
Nov 02 10:10:36.437066 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:36.436961    4449 scope.go:122] "RemoveContainer" containerID="ccebd4ead288c9ff8c987d9bdcf917c9f36659796952aa00039bd27a0c52a8c3"
Nov 02 10:10:36.449462 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:36.449437    4449 scope.go:122] "RemoveContainer" containerID="ccebd4ead288c9ff8c987d9bdcf917c9f36659796952aa00039bd27a0c52a8c3"
Nov 02 10:10:36.449831 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:10:36.449798    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccebd4ead288c9ff8c987d9bdcf917c9f36659796952aa00039bd27a0c52a8c3\": container with ID starting with ccebd4ead288c9ff8c987d9bdcf917c9f36659796952aa00039bd27a0c52a8c3 not found: ID does not exist" containerID="ccebd4ead288c9ff8c987d9bdcf917c9f36659796952aa00039bd27a0c52a8c3"
Nov 02 10:10:36.449937 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:36.449834    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccebd4ead288c9ff8c987d9bdcf917c9f36659796952aa00039bd27a0c52a8c3"} err="failed to get container status \"ccebd4ead288c9ff8c987d9bdcf917c9f36659796952aa00039bd27a0c52a8c3\": rpc error: code = NotFound desc = could not find container \"ccebd4ead288c9ff8c987d9bdcf917c9f36659796952aa00039bd27a0c52a8c3\": container with ID starting with ccebd4ead288c9ff8c987d9bdcf917c9f36659796952aa00039bd27a0c52a8c3 not found: ID does not exist"
Nov 02 10:10:36.466159 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:36.466126    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"]
Nov 02 10:10:36.469883 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:36.469855    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-dd40e2/simulator-single-main-dd40e2-kserve-77b8557869-4wg4t"]
Nov 02 10:10:36.546625 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:10:36.546587    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:10:37.206703 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:37.206669    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08fb94e1-62d9-4dc6-b871-5e72800e4e84" path="/var/lib/kubelet/pods/08fb94e1-62d9-4dc6-b871-5e72800e4e84/volumes"
Nov 02 10:10:44.326447 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:44.326405    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"]
Nov 02 10:10:44.326932 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:44.326663    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy" containerID="cri-o://54a36bb542298db3cdf2852fef4d5bcd18155307fb4acbb2f0176a7c00857713" gracePeriod=30
Nov 02 10:10:45.964888 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:45.964845    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7" podUID="32e6ee34-2b8b-471f-88c9-87eb537d53ea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:10:45.965422 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:45.964853    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl" podUID="f4b1dac6-6095-4c85-ae74-f814e883d9ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:10:46.445984 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.445954    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"
Nov 02 10:10:46.449024 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.449000    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"
Nov 02 10:10:46.499948 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.499911    4449 generic.go:356] "Generic (PLEG): container finished" podID="32e6ee34-2b8b-471f-88c9-87eb537d53ea" containerID="7e8fff517f5965675eb69774307fb56c611e3def64b7942385f0a4a9971d27b3" exitCode=0
Nov 02 10:10:46.500131 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.499957    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"
Nov 02 10:10:46.500131 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.499989    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7" event={"ID":"32e6ee34-2b8b-471f-88c9-87eb537d53ea","Type":"ContainerDied","Data":"7e8fff517f5965675eb69774307fb56c611e3def64b7942385f0a4a9971d27b3"}
Nov 02 10:10:46.500131 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.500046    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7" event={"ID":"32e6ee34-2b8b-471f-88c9-87eb537d53ea","Type":"ContainerDied","Data":"ee6f5d727d62188589baa46dbbf3174a843eb6f62fc7bf3e4b6a1b50da1fa73a"}
Nov 02 10:10:46.500131 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.500080    4449 scope.go:122] "RemoveContainer" containerID="7e8fff517f5965675eb69774307fb56c611e3def64b7942385f0a4a9971d27b3"
Nov 02 10:10:46.501521 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.501497    4449 generic.go:356] "Generic (PLEG): container finished" podID="f4b1dac6-6095-4c85-ae74-f814e883d9ce" containerID="e664ad9d95c4a0bbcf1407ec3bbf5de24b9a5fdd220bbfced40e46fd78b680e0" exitCode=0
Nov 02 10:10:46.501660 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.501535    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"
Nov 02 10:10:46.501660 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.501577    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl" event={"ID":"f4b1dac6-6095-4c85-ae74-f814e883d9ce","Type":"ContainerDied","Data":"e664ad9d95c4a0bbcf1407ec3bbf5de24b9a5fdd220bbfced40e46fd78b680e0"}
Nov 02 10:10:46.501660 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.501615    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl" event={"ID":"f4b1dac6-6095-4c85-ae74-f814e883d9ce","Type":"ContainerDied","Data":"be28b6492d5d197dbc88e828b766e214210665c2cb5dc46655e19e4e376074e2"}
Nov 02 10:10:46.510020 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.509995    4449 scope.go:122] "RemoveContainer" containerID="7e8fff517f5965675eb69774307fb56c611e3def64b7942385f0a4a9971d27b3"
Nov 02 10:10:46.510416 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:10:46.510396    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e8fff517f5965675eb69774307fb56c611e3def64b7942385f0a4a9971d27b3\": container with ID starting with 7e8fff517f5965675eb69774307fb56c611e3def64b7942385f0a4a9971d27b3 not found: ID does not exist" containerID="7e8fff517f5965675eb69774307fb56c611e3def64b7942385f0a4a9971d27b3"
Nov 02 10:10:46.510519 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.510423    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8fff517f5965675eb69774307fb56c611e3def64b7942385f0a4a9971d27b3"} err="failed to get container status \"7e8fff517f5965675eb69774307fb56c611e3def64b7942385f0a4a9971d27b3\": rpc error: code = NotFound desc = could not find container \"7e8fff517f5965675eb69774307fb56c611e3def64b7942385f0a4a9971d27b3\": container with ID starting with 7e8fff517f5965675eb69774307fb56c611e3def64b7942385f0a4a9971d27b3 not found: ID does not exist"
Nov 02 10:10:46.510519 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.510442    4449 scope.go:122] "RemoveContainer" containerID="e664ad9d95c4a0bbcf1407ec3bbf5de24b9a5fdd220bbfced40e46fd78b680e0"
Nov 02 10:10:46.519677 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.519653    4449 scope.go:122] "RemoveContainer" containerID="e664ad9d95c4a0bbcf1407ec3bbf5de24b9a5fdd220bbfced40e46fd78b680e0"
Nov 02 10:10:46.520006 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:10:46.519980    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e664ad9d95c4a0bbcf1407ec3bbf5de24b9a5fdd220bbfced40e46fd78b680e0\": container with ID starting with e664ad9d95c4a0bbcf1407ec3bbf5de24b9a5fdd220bbfced40e46fd78b680e0 not found: ID does not exist" containerID="e664ad9d95c4a0bbcf1407ec3bbf5de24b9a5fdd220bbfced40e46fd78b680e0"
Nov 02 10:10:46.520093 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.520017    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e664ad9d95c4a0bbcf1407ec3bbf5de24b9a5fdd220bbfced40e46fd78b680e0"} err="failed to get container status \"e664ad9d95c4a0bbcf1407ec3bbf5de24b9a5fdd220bbfced40e46fd78b680e0\": rpc error: code = NotFound desc = could not find container \"e664ad9d95c4a0bbcf1407ec3bbf5de24b9a5fdd220bbfced40e46fd78b680e0\": container with ID starting with e664ad9d95c4a0bbcf1407ec3bbf5de24b9a5fdd220bbfced40e46fd78b680e0 not found: ID does not exist"
Nov 02 10:10:46.572494 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.572447    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e6ee34-2b8b-471f-88c9-87eb537d53ea-kube-api-access-sdsrs\" (UniqueName: \"kubernetes.io/projected/32e6ee34-2b8b-471f-88c9-87eb537d53ea-kube-api-access-sdsrs\") pod \"32e6ee34-2b8b-471f-88c9-87eb537d53ea\" (UID: \"32e6ee34-2b8b-471f-88c9-87eb537d53ea\") "
Nov 02 10:10:46.572683 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.572509    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b1dac6-6095-4c85-ae74-f814e883d9ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b1dac6-6095-4c85-ae74-f814e883d9ce-plugins-config-volume\") pod \"f4b1dac6-6095-4c85-ae74-f814e883d9ce\" (UID: \"f4b1dac6-6095-4c85-ae74-f814e883d9ce\") "
Nov 02 10:10:46.572683 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.572574    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b1dac6-6095-4c85-ae74-f814e883d9ce-kube-api-access-w77fk\" (UniqueName: \"kubernetes.io/projected/f4b1dac6-6095-4c85-ae74-f814e883d9ce-kube-api-access-w77fk\") pod \"f4b1dac6-6095-4c85-ae74-f814e883d9ce\" (UID: \"f4b1dac6-6095-4c85-ae74-f814e883d9ce\") "
Nov 02 10:10:46.572683 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.572674    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e6ee34-2b8b-471f-88c9-87eb537d53ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e6ee34-2b8b-471f-88c9-87eb537d53ea-plugins-config-volume\") pod \"32e6ee34-2b8b-471f-88c9-87eb537d53ea\" (UID: \"32e6ee34-2b8b-471f-88c9-87eb537d53ea\") "
Nov 02 10:10:46.572909 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.572886    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b1dac6-6095-4c85-ae74-f814e883d9ce-plugins-config-volume" pod "f4b1dac6-6095-4c85-ae74-f814e883d9ce" (UID: "f4b1dac6-6095-4c85-ae74-f814e883d9ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:10:46.573094 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.573071    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e6ee34-2b8b-471f-88c9-87eb537d53ea-plugins-config-volume" pod "32e6ee34-2b8b-471f-88c9-87eb537d53ea" (UID: "32e6ee34-2b8b-471f-88c9-87eb537d53ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:10:46.574987 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.574958    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e6ee34-2b8b-471f-88c9-87eb537d53ea-kube-api-access-sdsrs" pod "32e6ee34-2b8b-471f-88c9-87eb537d53ea" (UID: "32e6ee34-2b8b-471f-88c9-87eb537d53ea"). InnerVolumeSpecName "kube-api-access-sdsrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:10:46.575118 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.575032    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b1dac6-6095-4c85-ae74-f814e883d9ce-kube-api-access-w77fk" pod "f4b1dac6-6095-4c85-ae74-f814e883d9ce" (UID: "f4b1dac6-6095-4c85-ae74-f814e883d9ce"). InnerVolumeSpecName "kube-api-access-w77fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:10:46.674560 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.674522    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdsrs\" (UniqueName: \"kubernetes.io/projected/32e6ee34-2b8b-471f-88c9-87eb537d53ea-kube-api-access-sdsrs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:46.674560 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.674555    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b1dac6-6095-4c85-ae74-f814e883d9ce-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:46.674560 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.674566    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w77fk\" (UniqueName: \"kubernetes.io/projected/f4b1dac6-6095-4c85-ae74-f814e883d9ce-kube-api-access-w77fk\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:46.674800 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.674576    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e6ee34-2b8b-471f-88c9-87eb537d53ea-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:46.827851 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.827814    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"]
Nov 02 10:10:46.834939 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.834892    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-dd40e2-6d5cdm2dt7"]
Nov 02 10:10:46.844704 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.844674    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"]
Nov 02 10:10:46.851048 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:46.851014    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dd40e2-d69d5d5d59rdvl"]
Nov 02 10:10:47.205053 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:47.205020    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e6ee34-2b8b-471f-88c9-87eb537d53ea" path="/var/lib/kubelet/pods/32e6ee34-2b8b-471f-88c9-87eb537d53ea/volumes"
Nov 02 10:10:47.205463 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:47.205449    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b1dac6-6095-4c85-ae74-f814e883d9ce" path="/var/lib/kubelet/pods/f4b1dac6-6095-4c85-ae74-f814e883d9ce/volumes"
Nov 02 10:10:48.417843 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.417803    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"]
Nov 02 10:10:48.418336 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.418321    4449 cpu_manager.go:414] "containerMap: removing container" podUID="32e6ee34-2b8b-471f-88c9-87eb537d53ea" containerName="payload-pre-processing"
Nov 02 10:10:48.418387 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.418337    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e6ee34-2b8b-471f-88c9-87eb537d53ea" containerName="payload-pre-processing"
Nov 02 10:10:48.418387 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.418351    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f4b1dac6-6095-4c85-ae74-f814e883d9ce" containerName="payload-processing"
Nov 02 10:10:48.418387 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.418356    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b1dac6-6095-4c85-ae74-f814e883d9ce" containerName="payload-processing"
Nov 02 10:10:48.418387 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.418365    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a7030264-16be-48b6-ba8d-d526df941d82" containerName="revoke-keys"
Nov 02 10:10:48.418387 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.418371    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7030264-16be-48b6-ba8d-d526df941d82" containerName="revoke-keys"
Nov 02 10:10:48.418387 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.418387    4449 cpu_manager.go:414] "containerMap: removing container" podUID="08fb94e1-62d9-4dc6-b871-5e72800e4e84" containerName="main"
Nov 02 10:10:48.418576 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.418392    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fb94e1-62d9-4dc6-b871-5e72800e4e84" containerName="main"
Nov 02 10:10:48.418576 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.418452    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e6ee34-2b8b-471f-88c9-87eb537d53ea" containerName="payload-pre-processing"
Nov 02 10:10:48.418576 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.418460    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b1dac6-6095-4c85-ae74-f814e883d9ce" containerName="payload-processing"
Nov 02 10:10:48.418576 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.418466    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7030264-16be-48b6-ba8d-d526df941d82" containerName="revoke-keys"
Nov 02 10:10:48.418576 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.418474    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fb94e1-62d9-4dc6-b871-5e72800e4e84" containerName="main"
Nov 02 10:10:48.421880 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.421863    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.430021 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.429998    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bd1362d4-kserve-self-signed-certs\""
Nov 02 10:10:48.433564 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.433531    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"]
Nov 02 10:10:48.492655 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.492622    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-dshm\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.492812 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.492679    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-tmp-dir\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.492919 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.492900    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-home\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.492956 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.492931    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72qbl\" (UniqueName: \"kubernetes.io/projected/5fab759f-9c90-49f5-8931-46ddc8777829-kube-api-access-72qbl\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.492994 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.492965    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fab759f-9c90-49f5-8931-46ddc8777829-tls-certs\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.493058 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.493044    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-model-cache\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.593665 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.593626    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-home\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.593665 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.593666    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qbl\" (UniqueName: \"kubernetes.io/projected/5fab759f-9c90-49f5-8931-46ddc8777829-kube-api-access-72qbl\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.593923 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.593699    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fab759f-9c90-49f5-8931-46ddc8777829-tls-certs\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.593923 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.593741    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-model-cache\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.593923 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.593840    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-dshm\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.593923 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.593899    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-tmp-dir\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.594241 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.594112    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-home\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.594302 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.594249    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-model-cache\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.594366 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.594319    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-tmp-dir\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.596474 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.596442    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-dshm\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.596668 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.596650    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fab759f-9c90-49f5-8931-46ddc8777829-tls-certs\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.607788 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.607760    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72qbl\" (UniqueName: \"kubernetes.io/projected/5fab759f-9c90-49f5-8931-46ddc8777829-kube-api-access-72qbl\") pod \"e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") " pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:48.734197 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:48.734113    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:49.015707 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.015679    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"]
Nov 02 10:10:49.016832 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:10:49.016804    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fab759f_9c90_49f5_8931_46ddc8777829.slice/crio-021ce54ec63625502ead85214a69b5994bddeb6d86c15d75233ffd8afc72ce52 WatchSource:0}: Error finding container 021ce54ec63625502ead85214a69b5994bddeb6d86c15d75233ffd8afc72ce52: Status 404 returned error can't find the container with id 021ce54ec63625502ead85214a69b5994bddeb6d86c15d75233ffd8afc72ce52
Nov 02 10:10:49.526270 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.526189    4449 generic.go:356] "Generic (PLEG): container finished" podID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerID="54a36bb542298db3cdf2852fef4d5bcd18155307fb4acbb2f0176a7c00857713" exitCode=0
Nov 02 10:10:49.526270 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.526249    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" event={"ID":"de9cd89b-23d7-4cb2-a020-35ea8c107d81","Type":"ContainerDied","Data":"54a36bb542298db3cdf2852fef4d5bcd18155307fb4acbb2f0176a7c00857713"}
Nov 02 10:10:49.526781 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.526326    4449 scope.go:122] "RemoveContainer" containerID="ad65e14cc23cfdded878b38b82973442ac17f2cf22302f00e27f83d33fe9615e"
Nov 02 10:10:49.528411 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.528380    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8" event={"ID":"5fab759f-9c90-49f5-8931-46ddc8777829","Type":"ContainerStarted","Data":"9929d7e820121c33099c77e26f32bf8cf08583d3a81ed4f9355238727853ff11"}
Nov 02 10:10:49.528570 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.528426    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8" event={"ID":"5fab759f-9c90-49f5-8931-46ddc8777829","Type":"ContainerStarted","Data":"021ce54ec63625502ead85214a69b5994bddeb6d86c15d75233ffd8afc72ce52"}
Nov 02 10:10:49.550019 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.549965    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8" podStartSLOduration=1.5499516309999999 podStartE2EDuration="1.549951631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:10:49.54718099 +0000 UTC m=+2126.485467787" watchObservedRunningTime="2026-11-02 10:10:49.549951631 +0000 UTC m=+2126.488238400"
Nov 02 10:10:49.574490 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.574463    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:10:49.708526 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.708496    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-socket\") pod \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") "
Nov 02 10:10:49.708708 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.708550    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-credential-socket\") pod \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") "
Nov 02 10:10:49.708708 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.708575    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-envoy\") pod \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") "
Nov 02 10:10:49.708708 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.708603    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-data\") pod \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") "
Nov 02 10:10:49.708708 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.708624    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istiod-ca-cert\") pod \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") "
Nov 02 10:10:49.708708 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.708662    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-podinfo\") pod \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") "
Nov 02 10:10:49.708708 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.708709    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-certs\") pod \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") "
Nov 02 10:10:49.709028 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.708738    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-kube-api-access-x44kq\" (UniqueName: \"kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-kube-api-access-x44kq\") pod \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") "
Nov 02 10:10:49.709028 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.708784    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-token\" (UniqueName: \"kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-token\") pod \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\" (UID: \"de9cd89b-23d7-4cb2-a020-35ea8c107d81\") "
Nov 02 10:10:49.709028 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.708818    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-socket" pod "de9cd89b-23d7-4cb2-a020-35ea8c107d81" (UID: "de9cd89b-23d7-4cb2-a020-35ea8c107d81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:10:49.709028 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.708985    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-credential-socket" pod "de9cd89b-23d7-4cb2-a020-35ea8c107d81" (UID: "de9cd89b-23d7-4cb2-a020-35ea8c107d81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:10:49.709274 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.709061    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-certs" pod "de9cd89b-23d7-4cb2-a020-35ea8c107d81" (UID: "de9cd89b-23d7-4cb2-a020-35ea8c107d81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:10:49.709274 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.709085    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istiod-ca-cert" pod "de9cd89b-23d7-4cb2-a020-35ea8c107d81" (UID: "de9cd89b-23d7-4cb2-a020-35ea8c107d81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:10:49.709274 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.709200    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:49.709448 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.709338    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:49.709448 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.709353    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:49.709448 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.709361    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:49.709448 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.709296    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-data" pod "de9cd89b-23d7-4cb2-a020-35ea8c107d81" (UID: "de9cd89b-23d7-4cb2-a020-35ea8c107d81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:10:49.711192 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.711162    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-kube-api-access-x44kq" pod "de9cd89b-23d7-4cb2-a020-35ea8c107d81" (UID: "de9cd89b-23d7-4cb2-a020-35ea8c107d81"). InnerVolumeSpecName "kube-api-access-x44kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:10:49.711362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.711188    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-token" pod "de9cd89b-23d7-4cb2-a020-35ea8c107d81" (UID: "de9cd89b-23d7-4cb2-a020-35ea8c107d81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:10:49.711501 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.711485    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-podinfo" pod "de9cd89b-23d7-4cb2-a020-35ea8c107d81" (UID: "de9cd89b-23d7-4cb2-a020-35ea8c107d81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:10:49.711558 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.711496    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-envoy" pod "de9cd89b-23d7-4cb2-a020-35ea8c107d81" (UID: "de9cd89b-23d7-4cb2-a020-35ea8c107d81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:10:49.734281 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.734251    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:49.811311 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.811277    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:49.811311 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.811305    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:49.811311 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.811316    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:49.811554 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.811326    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x44kq\" (UniqueName: \"kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-kube-api-access-x44kq\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:49.811554 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:49.811336    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9cd89b-23d7-4cb2-a020-35ea8c107d81-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:50.541242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:50.540326    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"
Nov 02 10:10:50.541242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:50.540319    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68" event={"ID":"de9cd89b-23d7-4cb2-a020-35ea8c107d81","Type":"ContainerDied","Data":"2054cf7c4718240be444b073c9866d18a51ada2526a0895c1a6703b5af448513"}
Nov 02 10:10:50.541242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:50.540545    4449 scope.go:122] "RemoveContainer" containerID="54a36bb542298db3cdf2852fef4d5bcd18155307fb4acbb2f0176a7c00857713"
Nov 02 10:10:50.573378 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:50.573339    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"]
Nov 02 10:10:50.580937 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:50.580878    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dd40e2-openshift-default-5fd56bbf97qkv68"]
Nov 02 10:10:51.223298 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:51.223255    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" path="/var/lib/kubelet/pods/de9cd89b-23d7-4cb2-a020-35ea8c107d81/volumes"
Nov 02 10:10:58.734308 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:58.734263    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:58.747625 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:58.747595    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:10:59.636787 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:10:59.636748    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:11:06.546250 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:11:06.546187    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:11:22.349623 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.349584    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"]
Nov 02 10:11:22.350280 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.350261    4449 cpu_manager.go:414] "containerMap: removing container" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy"
Nov 02 10:11:22.350369 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.350283    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy"
Nov 02 10:11:22.350473 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.350458    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy"
Nov 02 10:11:22.353654 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.353631    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"
Nov 02 10:11:22.360502 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.360384    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:11:22.360502 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.360423    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:11:22.360502 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.360456    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:11:22.360854 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.360638    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:11:22.363098 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.363071    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"]
Nov 02 10:11:22.453268 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.453195    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4hqp9\" (UID: \"12dea4ac-9dd2-483b-b57b-285ef0a56592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"
Nov 02 10:11:22.453481 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.453331    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtgsb\" (UniqueName: \"kubernetes.io/projected/12dea4ac-9dd2-483b-b57b-285ef0a56592-kube-api-access-jtgsb\") pod \"maas-api-revoke-sub-452b6d7b-4hqp9\" (UID: \"12dea4ac-9dd2-483b-b57b-285ef0a56592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"
Nov 02 10:11:22.453481 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.453369    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4hqp9\" (UID: \"12dea4ac-9dd2-483b-b57b-285ef0a56592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"
Nov 02 10:11:22.555029 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.554987    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4hqp9\" (UID: \"12dea4ac-9dd2-483b-b57b-285ef0a56592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"
Nov 02 10:11:22.555029 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.555036    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtgsb\" (UniqueName: \"kubernetes.io/projected/12dea4ac-9dd2-483b-b57b-285ef0a56592-kube-api-access-jtgsb\") pod \"maas-api-revoke-sub-452b6d7b-4hqp9\" (UID: \"12dea4ac-9dd2-483b-b57b-285ef0a56592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"
Nov 02 10:11:22.555385 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.555064    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4hqp9\" (UID: \"12dea4ac-9dd2-483b-b57b-285ef0a56592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"
Nov 02 10:11:22.555817 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.555793    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4hqp9\" (UID: \"12dea4ac-9dd2-483b-b57b-285ef0a56592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"
Nov 02 10:11:22.557716 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.557697    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4hqp9\" (UID: \"12dea4ac-9dd2-483b-b57b-285ef0a56592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"
Nov 02 10:11:22.568071 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.568044    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtgsb\" (UniqueName: \"kubernetes.io/projected/12dea4ac-9dd2-483b-b57b-285ef0a56592-kube-api-access-jtgsb\") pod \"maas-api-revoke-sub-452b6d7b-4hqp9\" (UID: \"12dea4ac-9dd2-483b-b57b-285ef0a56592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"
Nov 02 10:11:22.670205 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.670120    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"
Nov 02 10:11:22.949574 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:22.949546    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"]
Nov 02 10:11:22.950832 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:11:22.950804    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12dea4ac_9dd2_483b_b57b_285ef0a56592.slice/crio-5cbbdecc3949517b946d01e7df57b2bd5231e644ec003cb8f0bcf8aca2dd0d9b WatchSource:0}: Error finding container 5cbbdecc3949517b946d01e7df57b2bd5231e644ec003cb8f0bcf8aca2dd0d9b: Status 404 returned error can't find the container with id 5cbbdecc3949517b946d01e7df57b2bd5231e644ec003cb8f0bcf8aca2dd0d9b
Nov 02 10:11:23.805171 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:23.805133    4449 generic.go:356] "Generic (PLEG): container finished" podID="12dea4ac-9dd2-483b-b57b-285ef0a56592" containerID="beef61f35b2475a36a899362afb7c835c733f1c842357a12e52301388f15fe31" exitCode=0
Nov 02 10:11:23.805748 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:23.805252    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9" event={"ID":"12dea4ac-9dd2-483b-b57b-285ef0a56592","Type":"ContainerDied","Data":"beef61f35b2475a36a899362afb7c835c733f1c842357a12e52301388f15fe31"}
Nov 02 10:11:23.805748 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:23.805321    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9" event={"ID":"12dea4ac-9dd2-483b-b57b-285ef0a56592","Type":"ContainerStarted","Data":"5cbbdecc3949517b946d01e7df57b2bd5231e644ec003cb8f0bcf8aca2dd0d9b"}
Nov 02 10:11:25.242195 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:25.242168    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"
Nov 02 10:11:25.337840 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:25.337801    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-xks-ca\") pod \"12dea4ac-9dd2-483b-b57b-285ef0a56592\" (UID: \"12dea4ac-9dd2-483b-b57b-285ef0a56592\") "
Nov 02 10:11:25.338012 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:25.337848    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-openshift-ca\") pod \"12dea4ac-9dd2-483b-b57b-285ef0a56592\" (UID: \"12dea4ac-9dd2-483b-b57b-285ef0a56592\") "
Nov 02 10:11:25.338012 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:25.337894    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12dea4ac-9dd2-483b-b57b-285ef0a56592-kube-api-access-jtgsb\" (UniqueName: \"kubernetes.io/projected/12dea4ac-9dd2-483b-b57b-285ef0a56592-kube-api-access-jtgsb\") pod \"12dea4ac-9dd2-483b-b57b-285ef0a56592\" (UID: \"12dea4ac-9dd2-483b-b57b-285ef0a56592\") "
Nov 02 10:11:25.338430 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:25.338401    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-openshift-ca" pod "12dea4ac-9dd2-483b-b57b-285ef0a56592" (UID: "12dea4ac-9dd2-483b-b57b-285ef0a56592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:11:25.340170 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:25.340134    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-xks-ca" pod "12dea4ac-9dd2-483b-b57b-285ef0a56592" (UID: "12dea4ac-9dd2-483b-b57b-285ef0a56592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:11:25.340423 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:25.340403    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12dea4ac-9dd2-483b-b57b-285ef0a56592-kube-api-access-jtgsb" pod "12dea4ac-9dd2-483b-b57b-285ef0a56592" (UID: "12dea4ac-9dd2-483b-b57b-285ef0a56592"). InnerVolumeSpecName "kube-api-access-jtgsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:11:25.439095 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:25.439005    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtgsb\" (UniqueName: \"kubernetes.io/projected/12dea4ac-9dd2-483b-b57b-285ef0a56592-kube-api-access-jtgsb\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:11:25.439095 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:25.439041    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:11:25.439095 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:25.439051    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12dea4ac-9dd2-483b-b57b-285ef0a56592-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:11:25.845078 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:25.845044    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9" event={"ID":"12dea4ac-9dd2-483b-b57b-285ef0a56592","Type":"ContainerDied","Data":"5cbbdecc3949517b946d01e7df57b2bd5231e644ec003cb8f0bcf8aca2dd0d9b"}
Nov 02 10:11:25.845281 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:25.845093    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cbbdecc3949517b946d01e7df57b2bd5231e644ec003cb8f0bcf8aca2dd0d9b"
Nov 02 10:11:25.845281 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:25.845051    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"
Nov 02 10:11:33.217545 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:33.217508    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"]
Nov 02 10:11:36.633242 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:11:36.632398    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:11:39.620587 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.620548    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"]
Nov 02 10:11:39.621207 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.621185    4449 cpu_manager.go:414] "containerMap: removing container" podUID="12dea4ac-9dd2-483b-b57b-285ef0a56592" containerName="revoke-keys"
Nov 02 10:11:39.621324 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.621210    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12dea4ac-9dd2-483b-b57b-285ef0a56592" containerName="revoke-keys"
Nov 02 10:11:39.621324 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.621244    4449 cpu_manager.go:414] "containerMap: removing container" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy"
Nov 02 10:11:39.621324 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.621251    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy"
Nov 02 10:11:39.621486 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.621420    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="12dea4ac-9dd2-483b-b57b-285ef0a56592" containerName="revoke-keys"
Nov 02 10:11:39.621486 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.621440    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9cd89b-23d7-4cb2-a020-35ea8c107d81" containerName="istio-proxy"
Nov 02 10:11:39.626356 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.626324    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"
Nov 02 10:11:39.635730 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.635688    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"]
Nov 02 10:11:39.636388 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.636351    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:11:39.638398 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.637089    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:11:39.638398 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.637170    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:11:39.638398 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.637265    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:11:39.741156 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.741106    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29484203-c406-4f33-b328-0162e1472972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-4rfzs\" (UID: \"29484203-c406-4f33-b328-0162e1472972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"
Nov 02 10:11:39.741392 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.741196    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkr9g\" (UniqueName: \"kubernetes.io/projected/29484203-c406-4f33-b328-0162e1472972-kube-api-access-vkr9g\") pod \"maas-api-revoke-sub-3e31a506-4rfzs\" (UID: \"29484203-c406-4f33-b328-0162e1472972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"
Nov 02 10:11:39.741476 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.741413    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29484203-c406-4f33-b328-0162e1472972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-4rfzs\" (UID: \"29484203-c406-4f33-b328-0162e1472972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"
Nov 02 10:11:39.842390 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.842340    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29484203-c406-4f33-b328-0162e1472972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-4rfzs\" (UID: \"29484203-c406-4f33-b328-0162e1472972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"
Nov 02 10:11:39.842599 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.842463    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkr9g\" (UniqueName: \"kubernetes.io/projected/29484203-c406-4f33-b328-0162e1472972-kube-api-access-vkr9g\") pod \"maas-api-revoke-sub-3e31a506-4rfzs\" (UID: \"29484203-c406-4f33-b328-0162e1472972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"
Nov 02 10:11:39.842599 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.842585    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29484203-c406-4f33-b328-0162e1472972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-4rfzs\" (UID: \"29484203-c406-4f33-b328-0162e1472972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"
Nov 02 10:11:39.843474 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.843446    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29484203-c406-4f33-b328-0162e1472972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-4rfzs\" (UID: \"29484203-c406-4f33-b328-0162e1472972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"
Nov 02 10:11:39.845156 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.845130    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29484203-c406-4f33-b328-0162e1472972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-4rfzs\" (UID: \"29484203-c406-4f33-b328-0162e1472972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"
Nov 02 10:11:39.856299 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.856268    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkr9g\" (UniqueName: \"kubernetes.io/projected/29484203-c406-4f33-b328-0162e1472972-kube-api-access-vkr9g\") pod \"maas-api-revoke-sub-3e31a506-4rfzs\" (UID: \"29484203-c406-4f33-b328-0162e1472972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"
Nov 02 10:11:39.954692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:39.954597    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"
Nov 02 10:11:40.249052 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:40.248970    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"]
Nov 02 10:11:40.253320 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:11:40.253287    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29484203_c406_4f33_b328_0162e1472972.slice/crio-e69abe30526bac55e4a151f08f1fd86482072ee90dd23b22f7c109c74f5d6de2 WatchSource:0}: Error finding container e69abe30526bac55e4a151f08f1fd86482072ee90dd23b22f7c109c74f5d6de2: Status 404 returned error can't find the container with id e69abe30526bac55e4a151f08f1fd86482072ee90dd23b22f7c109c74f5d6de2
Nov 02 10:11:41.022963 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:41.022854    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs" event={"ID":"29484203-c406-4f33-b328-0162e1472972","Type":"ContainerStarted","Data":"052b99e01f0e850448eda14a3d54e1ebd123d845494e0f619e7eb277117713cd"}
Nov 02 10:11:41.023501 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:41.022967    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs" event={"ID":"29484203-c406-4f33-b328-0162e1472972","Type":"ContainerStarted","Data":"e69abe30526bac55e4a151f08f1fd86482072ee90dd23b22f7c109c74f5d6de2"}
Nov 02 10:11:41.045078 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:41.045024    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs" podStartSLOduration=2.04501129 podStartE2EDuration="2.04501129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:11:41.042917281 +0000 UTC m=+2177.981204057" watchObservedRunningTime="2026-11-02 10:11:41.04501129 +0000 UTC m=+2177.983298098"
Nov 02 10:11:42.033174 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:42.033135    4449 generic.go:356] "Generic (PLEG): container finished" podID="29484203-c406-4f33-b328-0162e1472972" containerID="052b99e01f0e850448eda14a3d54e1ebd123d845494e0f619e7eb277117713cd" exitCode=0
Nov 02 10:11:42.033602 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:42.033199    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs" event={"ID":"29484203-c406-4f33-b328-0162e1472972","Type":"ContainerDied","Data":"052b99e01f0e850448eda14a3d54e1ebd123d845494e0f619e7eb277117713cd"}
Nov 02 10:11:43.195939 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:43.195913    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"
Nov 02 10:11:43.337973 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:43.337894    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29484203-c406-4f33-b328-0162e1472972-kube-api-access-vkr9g\" (UniqueName: \"kubernetes.io/projected/29484203-c406-4f33-b328-0162e1472972-kube-api-access-vkr9g\") pod \"29484203-c406-4f33-b328-0162e1472972\" (UID: \"29484203-c406-4f33-b328-0162e1472972\") "
Nov 02 10:11:43.337973 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:43.337938    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29484203-c406-4f33-b328-0162e1472972-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29484203-c406-4f33-b328-0162e1472972-maas-api-openshift-ca\") pod \"29484203-c406-4f33-b328-0162e1472972\" (UID: \"29484203-c406-4f33-b328-0162e1472972\") "
Nov 02 10:11:43.338183 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:43.338107    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29484203-c406-4f33-b328-0162e1472972-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29484203-c406-4f33-b328-0162e1472972-maas-api-xks-ca\") pod \"29484203-c406-4f33-b328-0162e1472972\" (UID: \"29484203-c406-4f33-b328-0162e1472972\") "
Nov 02 10:11:43.338311 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:43.338284    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29484203-c406-4f33-b328-0162e1472972-maas-api-openshift-ca" pod "29484203-c406-4f33-b328-0162e1472972" (UID: "29484203-c406-4f33-b328-0162e1472972"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:11:43.338501 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:43.338485    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29484203-c406-4f33-b328-0162e1472972-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:11:43.340236 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:43.340198    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29484203-c406-4f33-b328-0162e1472972-kube-api-access-vkr9g" pod "29484203-c406-4f33-b328-0162e1472972" (UID: "29484203-c406-4f33-b328-0162e1472972"). InnerVolumeSpecName "kube-api-access-vkr9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:11:43.340425 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:43.340404    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29484203-c406-4f33-b328-0162e1472972-maas-api-xks-ca" pod "29484203-c406-4f33-b328-0162e1472972" (UID: "29484203-c406-4f33-b328-0162e1472972"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:11:43.439970 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:43.439925    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29484203-c406-4f33-b328-0162e1472972-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:11:43.439970 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:43.439964    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkr9g\" (UniqueName: \"kubernetes.io/projected/29484203-c406-4f33-b328-0162e1472972-kube-api-access-vkr9g\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:11:44.047202 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:44.047169    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs" event={"ID":"29484203-c406-4f33-b328-0162e1472972","Type":"ContainerDied","Data":"e69abe30526bac55e4a151f08f1fd86482072ee90dd23b22f7c109c74f5d6de2"}
Nov 02 10:11:44.047202 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:44.047188    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"
Nov 02 10:11:44.047472 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:44.047258    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e69abe30526bac55e4a151f08f1fd86482072ee90dd23b22f7c109c74f5d6de2"
Nov 02 10:11:48.238743 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.238695    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8" podUID="5fab759f-9c90-49f5-8931-46ddc8777829" containerName="main" containerID="cri-o://9929d7e820121c33099c77e26f32bf8cf08583d3a81ed4f9355238727853ff11" gracePeriod=45
Nov 02 10:11:48.502714 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.502649    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:11:48.584249 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.584195    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fab759f-9c90-49f5-8931-46ddc8777829-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fab759f-9c90-49f5-8931-46ddc8777829-tls-certs\") pod \"5fab759f-9c90-49f5-8931-46ddc8777829\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") "
Nov 02 10:11:48.584469 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.584302    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-tmp-dir\") pod \"5fab759f-9c90-49f5-8931-46ddc8777829\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") "
Nov 02 10:11:48.584469 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.584326    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-model-cache\") pod \"5fab759f-9c90-49f5-8931-46ddc8777829\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") "
Nov 02 10:11:48.584469 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.584344    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-home\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-home\") pod \"5fab759f-9c90-49f5-8931-46ddc8777829\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") "
Nov 02 10:11:48.584469 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.584455    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fab759f-9c90-49f5-8931-46ddc8777829-kube-api-access-72qbl\" (UniqueName: \"kubernetes.io/projected/5fab759f-9c90-49f5-8931-46ddc8777829-kube-api-access-72qbl\") pod \"5fab759f-9c90-49f5-8931-46ddc8777829\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") "
Nov 02 10:11:48.584778 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.584531    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-dshm\") pod \"5fab759f-9c90-49f5-8931-46ddc8777829\" (UID: \"5fab759f-9c90-49f5-8931-46ddc8777829\") "
Nov 02 10:11:48.584778 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.584626    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-model-cache" pod "5fab759f-9c90-49f5-8931-46ddc8777829" (UID: "5fab759f-9c90-49f5-8931-46ddc8777829"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:11:48.584778 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.584709    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-home" pod "5fab759f-9c90-49f5-8931-46ddc8777829" (UID: "5fab759f-9c90-49f5-8931-46ddc8777829"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:11:48.584930 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.584889    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:11:48.584930 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.584897    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-tmp-dir" pod "5fab759f-9c90-49f5-8931-46ddc8777829" (UID: "5fab759f-9c90-49f5-8931-46ddc8777829"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:11:48.584930 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.584910    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:11:48.586689 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.586655    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fab759f-9c90-49f5-8931-46ddc8777829-tls-certs" pod "5fab759f-9c90-49f5-8931-46ddc8777829" (UID: "5fab759f-9c90-49f5-8931-46ddc8777829"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:11:48.586861 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.586846    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-dshm" pod "5fab759f-9c90-49f5-8931-46ddc8777829" (UID: "5fab759f-9c90-49f5-8931-46ddc8777829"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:11:48.587013 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.586972    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fab759f-9c90-49f5-8931-46ddc8777829-kube-api-access-72qbl" pod "5fab759f-9c90-49f5-8931-46ddc8777829" (UID: "5fab759f-9c90-49f5-8931-46ddc8777829"). InnerVolumeSpecName "kube-api-access-72qbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:11:48.687464 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.687326    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72qbl\" (UniqueName: \"kubernetes.io/projected/5fab759f-9c90-49f5-8931-46ddc8777829-kube-api-access-72qbl\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:11:48.687682 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.687655    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:11:48.687731 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.687723    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fab759f-9c90-49f5-8931-46ddc8777829-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:11:48.687812 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:48.687740    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fab759f-9c90-49f5-8931-46ddc8777829-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:11:49.079034 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:49.078998    4449 generic.go:356] "Generic (PLEG): container finished" podID="5fab759f-9c90-49f5-8931-46ddc8777829" containerID="9929d7e820121c33099c77e26f32bf8cf08583d3a81ed4f9355238727853ff11" exitCode=0
Nov 02 10:11:49.079290 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:49.079044    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8" event={"ID":"5fab759f-9c90-49f5-8931-46ddc8777829","Type":"ContainerDied","Data":"9929d7e820121c33099c77e26f32bf8cf08583d3a81ed4f9355238727853ff11"}
Nov 02 10:11:49.079290 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:49.079067    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"
Nov 02 10:11:49.079290 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:49.079085    4449 scope.go:122] "RemoveContainer" containerID="9929d7e820121c33099c77e26f32bf8cf08583d3a81ed4f9355238727853ff11"
Nov 02 10:11:49.079290 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:49.079076    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8" event={"ID":"5fab759f-9c90-49f5-8931-46ddc8777829","Type":"ContainerDied","Data":"021ce54ec63625502ead85214a69b5994bddeb6d86c15d75233ffd8afc72ce52"}
Nov 02 10:11:49.089580 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:49.089558    4449 scope.go:122] "RemoveContainer" containerID="9929d7e820121c33099c77e26f32bf8cf08583d3a81ed4f9355238727853ff11"
Nov 02 10:11:49.089903 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:11:49.089883    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9929d7e820121c33099c77e26f32bf8cf08583d3a81ed4f9355238727853ff11\": container with ID starting with 9929d7e820121c33099c77e26f32bf8cf08583d3a81ed4f9355238727853ff11 not found: ID does not exist" containerID="9929d7e820121c33099c77e26f32bf8cf08583d3a81ed4f9355238727853ff11"
Nov 02 10:11:49.089965 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:49.089917    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9929d7e820121c33099c77e26f32bf8cf08583d3a81ed4f9355238727853ff11"} err="failed to get container status \"9929d7e820121c33099c77e26f32bf8cf08583d3a81ed4f9355238727853ff11\": rpc error: code = NotFound desc = could not find container \"9929d7e820121c33099c77e26f32bf8cf08583d3a81ed4f9355238727853ff11\": container with ID starting with 9929d7e820121c33099c77e26f32bf8cf08583d3a81ed4f9355238727853ff11 not found: ID does not exist"
Nov 02 10:11:49.103109 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:49.103069    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"]
Nov 02 10:11:49.107073 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:49.107041    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bd1362d4-kserve-5b987dd75c-5fgb8"]
Nov 02 10:11:49.205521 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:49.205494    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fab759f-9c90-49f5-8931-46ddc8777829" path="/var/lib/kubelet/pods/5fab759f-9c90-49f5-8931-46ddc8777829/volumes"
Nov 02 10:11:58.440890 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.440852    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"]
Nov 02 10:11:58.441398 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.441315    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5fab759f-9c90-49f5-8931-46ddc8777829" containerName="main"
Nov 02 10:11:58.441398 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.441326    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fab759f-9c90-49f5-8931-46ddc8777829" containerName="main"
Nov 02 10:11:58.441398 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.441351    4449 cpu_manager.go:414] "containerMap: removing container" podUID="29484203-c406-4f33-b328-0162e1472972" containerName="revoke-keys"
Nov 02 10:11:58.441398 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.441356    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29484203-c406-4f33-b328-0162e1472972" containerName="revoke-keys"
Nov 02 10:11:58.441547 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.441416    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="29484203-c406-4f33-b328-0162e1472972" containerName="revoke-keys"
Nov 02 10:11:58.441547 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.441429    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fab759f-9c90-49f5-8931-46ddc8777829" containerName="main"
Nov 02 10:11:58.446273 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.446251    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"
Nov 02 10:11:58.456529 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.456496    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:11:58.456686 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.456588    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:11:58.464722 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.464699    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:11:58.468780 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.468742    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:11:58.472780 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.472751    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"]
Nov 02 10:11:58.558907 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.558866    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkzvn\" (UniqueName: \"kubernetes.io/projected/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-kube-api-access-pkzvn\") pod \"maas-api-revoke-sub-2195d4e8-2l6ns\" (UID: \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"
Nov 02 10:11:58.558907 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.558924    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2195d4e8-2l6ns\" (UID: \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"
Nov 02 10:11:58.559149 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.559002    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2195d4e8-2l6ns\" (UID: \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"
Nov 02 10:11:58.660058 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.660018    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkzvn\" (UniqueName: \"kubernetes.io/projected/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-kube-api-access-pkzvn\") pod \"maas-api-revoke-sub-2195d4e8-2l6ns\" (UID: \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"
Nov 02 10:11:58.660331 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.660075    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2195d4e8-2l6ns\" (UID: \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"
Nov 02 10:11:58.660331 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.660128    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2195d4e8-2l6ns\" (UID: \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"
Nov 02 10:11:58.661201 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.661169    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2195d4e8-2l6ns\" (UID: \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"
Nov 02 10:11:58.663075 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.663054    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2195d4e8-2l6ns\" (UID: \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"
Nov 02 10:11:58.673846 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.673821    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkzvn\" (UniqueName: \"kubernetes.io/projected/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-kube-api-access-pkzvn\") pod \"maas-api-revoke-sub-2195d4e8-2l6ns\" (UID: \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"
Nov 02 10:11:58.760728 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:58.760640    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"
Nov 02 10:11:59.046757 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:59.046732    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"]
Nov 02 10:11:59.048102 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:11:59.048073    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25e45fb_a71f_4492_9dfa_5eb60e5a9f0f.slice/crio-9d71af3f078e08b407cc7645a6cc9ed35edaa066934f17f6c6f5a7f2f56efb29 WatchSource:0}: Error finding container 9d71af3f078e08b407cc7645a6cc9ed35edaa066934f17f6c6f5a7f2f56efb29: Status 404 returned error can't find the container with id 9d71af3f078e08b407cc7645a6cc9ed35edaa066934f17f6c6f5a7f2f56efb29
Nov 02 10:11:59.187774 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:11:59.187732    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns" event={"ID":"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f","Type":"ContainerStarted","Data":"9d71af3f078e08b407cc7645a6cc9ed35edaa066934f17f6c6f5a7f2f56efb29"}
Nov 02 10:12:00.195370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:00.195275    4449 generic.go:356] "Generic (PLEG): container finished" podID="a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f" containerID="663612cf145dce4fe544360f6867b4380752a4442cf9aa48126e2e1af02c8741" exitCode=0
Nov 02 10:12:00.195370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:00.195355    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns" event={"ID":"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f","Type":"ContainerDied","Data":"663612cf145dce4fe544360f6867b4380752a4442cf9aa48126e2e1af02c8741"}
Nov 02 10:12:01.342135 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:01.342107    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"
Nov 02 10:12:01.387928 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:01.387894    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-xks-ca\") pod \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\" (UID: \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\") "
Nov 02 10:12:01.387928 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:01.387937    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-openshift-ca\") pod \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\" (UID: \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\") "
Nov 02 10:12:01.388178 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:01.388036    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-kube-api-access-pkzvn\" (UniqueName: \"kubernetes.io/projected/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-kube-api-access-pkzvn\") pod \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\" (UID: \"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f\") "
Nov 02 10:12:01.388504 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:01.388471    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-openshift-ca" pod "a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f" (UID: "a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:12:01.390425 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:01.390399    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-kube-api-access-pkzvn" pod "a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f" (UID: "a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f"). InnerVolumeSpecName "kube-api-access-pkzvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:12:01.390789 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:01.390769    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-xks-ca" pod "a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f" (UID: "a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:12:01.489728 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:01.489643    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:12:01.489728 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:01.489673    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:12:01.489728 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:01.489684    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkzvn\" (UniqueName: \"kubernetes.io/projected/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f-kube-api-access-pkzvn\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:12:02.208055 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:02.208026    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"
Nov 02 10:12:02.208055 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:02.208038    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns" event={"ID":"a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f","Type":"ContainerDied","Data":"9d71af3f078e08b407cc7645a6cc9ed35edaa066934f17f6c6f5a7f2f56efb29"}
Nov 02 10:12:02.208321 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:02.208082    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d71af3f078e08b407cc7645a6cc9ed35edaa066934f17f6c6f5a7f2f56efb29"
Nov 02 10:12:06.547396 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:12:06.547308    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:12:11.608734 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.608695    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"]
Nov 02 10:12:11.609423 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.609402    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f" containerName="revoke-keys"
Nov 02 10:12:11.609469 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.609426    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f" containerName="revoke-keys"
Nov 02 10:12:11.609584 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.609573    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f" containerName="revoke-keys"
Nov 02 10:12:11.613551 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.613536    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.624995 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.624972    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"]
Nov 02 10:12:11.637828 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.637803    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2b5a18-openshift-default-dockercfg-x9ds5\""
Nov 02 10:12:11.702553 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.702516    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-podinfo\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.702553 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.702565    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-socket\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.702837 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.702593    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-data\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.702837 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.702749    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-token\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.702837 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.702805    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istiod-ca-cert\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.703105 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.702835    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlhzx\" (UniqueName: \"kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-kube-api-access-nlhzx\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.703105 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.702874    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-envoy\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.703105 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.702910    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-credential-socket\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.703105 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.703010    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-certs\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.804200 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.804162    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-certs\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.804444 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.804248    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-podinfo\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.804444 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.804274    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-socket\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.804444 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.804289    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-data\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.804444 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.804340    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-token\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.804444 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.804379    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istiod-ca-cert\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.804444 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.804407    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlhzx\" (UniqueName: \"kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-kube-api-access-nlhzx\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.804444 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.804440    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-envoy\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.804853 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.804480    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-credential-socket\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.804853 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.804683    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-certs\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.804853 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.804792    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-socket\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.805011 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.804935    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-data\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.805243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.805107    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-credential-socket\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.805396 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.805375    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istiod-ca-cert\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.806904 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.806872    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-envoy\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.807028 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.806993    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-podinfo\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.817583 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.817527    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-token\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.817873 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.817846    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlhzx\" (UniqueName: \"kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-kube-api-access-nlhzx\") pod \"e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") " pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:11.928643 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:11.928553    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:12.214930 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:12.214904    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"]
Nov 02 10:12:12.216068 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:12:12.216041    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabcc4858_2830_479c_9e88_1bde7a9e1a0a.slice/crio-29fb5f6dd2cf38c9e4323a7cfa8c2fb55b23a12ce94c55c38b34e1f55c3b1072 WatchSource:0}: Error finding container 29fb5f6dd2cf38c9e4323a7cfa8c2fb55b23a12ce94c55c38b34e1f55c3b1072: Status 404 returned error can't find the container with id 29fb5f6dd2cf38c9e4323a7cfa8c2fb55b23a12ce94c55c38b34e1f55c3b1072
Nov 02 10:12:12.218324 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:12.218276    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:12:12.218412 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:12.218364    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:12:12.218412 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:12.218406    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:12:12.265435 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:12.265411    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" event={"ID":"abcc4858-2830-479c-9e88-1bde7a9e1a0a","Type":"ContainerStarted","Data":"29fb5f6dd2cf38c9e4323a7cfa8c2fb55b23a12ce94c55c38b34e1f55c3b1072"}
Nov 02 10:12:13.272533 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:13.272493    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" event={"ID":"abcc4858-2830-479c-9e88-1bde7a9e1a0a","Type":"ContainerStarted","Data":"d4a153e9101c74d93a4466fd0c2cc66ad99016d7c45dcfef0419000ab66e578a"}
Nov 02 10:12:13.297322 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:13.297271    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" podStartSLOduration=2.297255817 podStartE2EDuration="2.297255817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:12:13.296352663 +0000 UTC m=+2210.234639463" watchObservedRunningTime="2026-11-02 10:12:13.297255817 +0000 UTC m=+2210.235542590"
Nov 02 10:12:13.929009 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:13.928967    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:13.934381 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:13.934351    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:14.278100 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:14.278018    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:14.278955 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:14.278937    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:21.780206 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:21.780142    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"]
Nov 02 10:12:21.783941 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:21.783916    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"
Nov 02 10:12:21.791872 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:21.791334    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2b5a18-dockercfg-9jz5r\""
Nov 02 10:12:21.794729 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:21.794702    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"]
Nov 02 10:12:21.800595 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:21.800562    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lj6t\" (UniqueName: \"kubernetes.io/projected/99ad889f-918e-4e43-b38c-b4ee2bff1825-kube-api-access-7lj6t\") pod \"payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9\" (UID: \"99ad889f-918e-4e43-b38c-b4ee2bff1825\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"
Nov 02 10:12:21.800695 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:21.800643    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ad889f-918e-4e43-b38c-b4ee2bff1825-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9\" (UID: \"99ad889f-918e-4e43-b38c-b4ee2bff1825\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"
Nov 02 10:12:21.882282 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:21.872503    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2b5a18\""
Nov 02 10:12:21.901436 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:21.901371    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lj6t\" (UniqueName: \"kubernetes.io/projected/99ad889f-918e-4e43-b38c-b4ee2bff1825-kube-api-access-7lj6t\") pod \"payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9\" (UID: \"99ad889f-918e-4e43-b38c-b4ee2bff1825\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"
Nov 02 10:12:21.902283 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:21.902255    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ad889f-918e-4e43-b38c-b4ee2bff1825-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9\" (UID: \"99ad889f-918e-4e43-b38c-b4ee2bff1825\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"
Nov 02 10:12:21.903838 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:21.903803    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ad889f-918e-4e43-b38c-b4ee2bff1825-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9\" (UID: \"99ad889f-918e-4e43-b38c-b4ee2bff1825\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"
Nov 02 10:12:21.915259 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:21.915156    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lj6t\" (UniqueName: \"kubernetes.io/projected/99ad889f-918e-4e43-b38c-b4ee2bff1825-kube-api-access-7lj6t\") pod \"payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9\" (UID: \"99ad889f-918e-4e43-b38c-b4ee2bff1825\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"
Nov 02 10:12:22.025136 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:22.025024    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"]
Nov 02 10:12:22.027803 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:22.027776    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-r4nh4"]
Nov 02 10:12:22.100657 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:22.100622    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"
Nov 02 10:12:22.405197 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:22.405163    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"]
Nov 02 10:12:22.406725 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:12:22.406694    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ad889f_918e_4e43_b38c_b4ee2bff1825.slice/crio-de13883aba7ead8498c181dd2769d3ce314df4b4ee75c9f6949e34289d4fe053 WatchSource:0}: Error finding container de13883aba7ead8498c181dd2769d3ce314df4b4ee75c9f6949e34289d4fe053: Status 404 returned error can't find the container with id de13883aba7ead8498c181dd2769d3ce314df4b4ee75c9f6949e34289d4fe053
Nov 02 10:12:22.408723 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:22.408705    4449 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 10:12:23.206496 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:23.206460    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ed65ac-2201-41bd-b714-03c5f8af454a" path="/var/lib/kubelet/pods/d1ed65ac-2201-41bd-b714-03c5f8af454a/volumes"
Nov 02 10:12:23.338316 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:23.338281    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9" event={"ID":"99ad889f-918e-4e43-b38c-b4ee2bff1825","Type":"ContainerStarted","Data":"de13883aba7ead8498c181dd2769d3ce314df4b4ee75c9f6949e34289d4fe053"}
Nov 02 10:12:25.381036 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:25.381000    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9" event={"ID":"99ad889f-918e-4e43-b38c-b4ee2bff1825","Type":"ContainerStarted","Data":"b27b274203105cb4ba70650b1fa9c1aa3234a577e6d4b66246f2b13cf17fc814"}
Nov 02 10:12:25.381598 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:25.381083    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"
Nov 02 10:12:25.401680 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:25.401620    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9" podStartSLOduration=1.8067378729999999 podStartE2EDuration="4.401600275s" totalImagesPullingTime="2.594862402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:12:25.398840078 +0000 UTC m=+2222.337126877" watchObservedRunningTime="2026-11-02 10:12:25.401600275 +0000 UTC m=+2222.339887055"
Nov 02 10:12:29.570342 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:29.570206    4449 patch_prober.go:29] interesting pod/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": read tcp 10.134.0.2:33492->10.134.0.253:15021: read: connection reset by peer" start-of-body=
Nov 02 10:12:29.570788 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:29.570396    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": read tcp 10.134.0.2:33492->10.134.0.253:15021: read: connection reset by peer"
Nov 02 10:12:30.414932 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:30.414896    4449 generic.go:356] "Generic (PLEG): container finished" podID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerID="d4a153e9101c74d93a4466fd0c2cc66ad99016d7c45dcfef0419000ab66e578a" exitCode=0
Nov 02 10:12:30.415148 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:30.414978    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" event={"ID":"abcc4858-2830-479c-9e88-1bde7a9e1a0a","Type":"ContainerDied","Data":"d4a153e9101c74d93a4466fd0c2cc66ad99016d7c45dcfef0419000ab66e578a"}
Nov 02 10:12:30.415480 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:30.415462    4449 scope.go:122] "RemoveContainer" containerID="d4a153e9101c74d93a4466fd0c2cc66ad99016d7c45dcfef0419000ab66e578a"
Nov 02 10:12:30.416753 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:30.416723    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:12:30.416810 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:30.416792    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:12:30.416857 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:30.416824    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:12:31.422929 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:31.422892    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" event={"ID":"abcc4858-2830-479c-9e88-1bde7a9e1a0a","Type":"ContainerStarted","Data":"68120099b422d73721e41109aa2531aea5bd962c484d338b3778e382e090983d"}
Nov 02 10:12:31.929454 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:31.929413    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:31.929768 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:31.929742    4449 patch_prober.go:29] interesting pod/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 02 10:12:31.929885 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:31.929798    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 02 10:12:32.929282 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:32.929247    4449 patch_prober.go:29] interesting pod/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 02 10:12:32.929691 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:32.929299    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 02 10:12:33.929677 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:33.929639    4449 patch_prober.go:29] interesting pod/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 02 10:12:33.930089 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:33.929694    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 02 10:12:34.929265 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:34.929202    4449 patch_prober.go:29] interesting pod/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 02 10:12:34.929490 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:34.929288    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 02 10:12:35.929760 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:35.929723    4449 patch_prober.go:29] interesting pod/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 02 10:12:35.930178 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:35.929787    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 02 10:12:36.390531 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:36.390496    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"
Nov 02 10:12:36.549865 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:12:36.549827    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:12:36.932831 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:36.932801    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:36.933292 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:36.933188    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:36.933975 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:36.933954    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:12:39.034980 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:39.034942    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"]
Nov 02 10:12:39.038124 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:39.038101    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bmspd"]
Nov 02 10:12:39.205735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:39.205693    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a01aa5ff-d7af-4d85-817b-4dbe9f94f600" path="/var/lib/kubelet/pods/a01aa5ff-d7af-4d85-817b-4dbe9f94f600/volumes"
Nov 02 10:12:42.608441 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.608403    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"]
Nov 02 10:12:42.617670 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.617640    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.621562 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.621531    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"]
Nov 02 10:12:42.622174 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.622150    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-836bba-openshift-default-dockercfg-vdfzv\""
Nov 02 10:12:42.740199 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.740161    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-socket\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.740432 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.740246    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-data\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.740432 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.740281    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00e3a95-1a6e-411e-b928-456a968e432e-istio-podinfo\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.740432 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.740339    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svnkd\" (UniqueName: \"kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-kube-api-access-svnkd\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.740432 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.740423    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-credential-socket\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.740643 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.740501    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-envoy\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.740643 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.740531    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-certs\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.740643 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.740624    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00e3a95-1a6e-411e-b928-456a968e432e-istiod-ca-cert\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.740754 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.740656    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-istio-token\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.845850 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.845810    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-credential-socket\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.846105 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.846089    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-envoy\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.846233 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.846202    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-certs\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.846427 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.846413    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00e3a95-1a6e-411e-b928-456a968e432e-istiod-ca-cert\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.846534 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.846522    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-istio-token\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.846631 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.846617    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-socket\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.846735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.846722    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-data\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.846834 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.846822    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00e3a95-1a6e-411e-b928-456a968e432e-istio-podinfo\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.846936 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.846923    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svnkd\" (UniqueName: \"kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-kube-api-access-svnkd\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.847655 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.847605    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-certs\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.848208 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.848184    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-data\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.848522 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.848503    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-credential-socket\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.851188 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.851157    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00e3a95-1a6e-411e-b928-456a968e432e-istiod-ca-cert\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.851535 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.851511    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-socket\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.863310 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.856787    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00e3a95-1a6e-411e-b928-456a968e432e-istio-podinfo\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.863310 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.861812    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-envoy\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.863503 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.863404    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svnkd\" (UniqueName: \"kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-kube-api-access-svnkd\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.864702 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.864649    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-istio-token\") pod \"e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") " pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:42.934793 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:42.934756    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:43.219188 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:43.219161    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"]
Nov 02 10:12:43.219779 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:12:43.219750    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00e3a95_1a6e_411e_b928_456a968e432e.slice/crio-98f142998cf93cac2b29a0e00e3a3342c7a69fb16eb3186d7fe9e47bfc85d95e WatchSource:0}: Error finding container 98f142998cf93cac2b29a0e00e3a3342c7a69fb16eb3186d7fe9e47bfc85d95e: Status 404 returned error can't find the container with id 98f142998cf93cac2b29a0e00e3a3342c7a69fb16eb3186d7fe9e47bfc85d95e
Nov 02 10:12:43.222200 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:43.222162    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:12:43.222319 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:43.222295    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:12:43.222372 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:43.222355    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:12:43.500557 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:43.500460    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq" event={"ID":"c00e3a95-1a6e-411e-b928-456a968e432e","Type":"ContainerStarted","Data":"f554a33b889b6ab966636a02828b65ace2029a0f21adf3bd8ffebf2008cda36a"}
Nov 02 10:12:43.500557 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:43.500519    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq" event={"ID":"c00e3a95-1a6e-411e-b928-456a968e432e","Type":"ContainerStarted","Data":"98f142998cf93cac2b29a0e00e3a3342c7a69fb16eb3186d7fe9e47bfc85d95e"}
Nov 02 10:12:43.530459 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:43.530410    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq" podStartSLOduration=1.5303960920000002 podStartE2EDuration="1.530396092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:12:43.526532933 +0000 UTC m=+2240.464819734" watchObservedRunningTime="2026-11-02 10:12:43.530396092 +0000 UTC m=+2240.468682867"
Nov 02 10:12:43.935096 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:43.935056    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:44.940710 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:44.940677    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:45.513797 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:45.513764    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:45.514739 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:45.514720    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:12:50.025695 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:50.025656    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"]
Nov 02 10:12:50.030888 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:50.030858    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-f8h9n"]
Nov 02 10:12:51.206330 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:51.206298    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c9980b-7544-48f0-abc1-e162aea11c26" path="/var/lib/kubelet/pods/13c9980b-7544-48f0-abc1-e162aea11c26/volumes"
Nov 02 10:12:52.779315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:52.777513    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"]
Nov 02 10:12:52.782581 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:52.782551    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"
Nov 02 10:12:52.788825 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:52.788797    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-836bba-dockercfg-ttp8k\""
Nov 02 10:12:52.789233 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:52.789190    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-836bba\""
Nov 02 10:12:52.792455 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:52.792422    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"]
Nov 02 10:12:52.902631 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:52.902585    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rsjn\" (UniqueName: \"kubernetes.io/projected/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-kube-api-access-4rsjn\") pod \"payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k\" (UID: \"0da7b91c-3ac1-46b9-a47f-347e8775cf4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"
Nov 02 10:12:52.903412 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:52.902999    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k\" (UID: \"0da7b91c-3ac1-46b9-a47f-347e8775cf4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"
Nov 02 10:12:53.005350 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:53.005319    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k\" (UID: \"0da7b91c-3ac1-46b9-a47f-347e8775cf4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"
Nov 02 10:12:53.005546 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:53.005419    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsjn\" (UniqueName: \"kubernetes.io/projected/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-kube-api-access-4rsjn\") pod \"payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k\" (UID: \"0da7b91c-3ac1-46b9-a47f-347e8775cf4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"
Nov 02 10:12:53.006167 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:53.006139    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k\" (UID: \"0da7b91c-3ac1-46b9-a47f-347e8775cf4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"
Nov 02 10:12:53.018815 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:53.018792    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rsjn\" (UniqueName: \"kubernetes.io/projected/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-kube-api-access-4rsjn\") pod \"payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k\" (UID: \"0da7b91c-3ac1-46b9-a47f-347e8775cf4b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"
Nov 02 10:12:53.100175 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:53.100141    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"
Nov 02 10:12:53.408629 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:53.408588    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"]
Nov 02 10:12:53.409628 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:12:53.409600    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da7b91c_3ac1_46b9_a47f_347e8775cf4b.slice/crio-23ae41b3f5d99ad19b03c5ba5d00f6883953ffd86429c9dcd611506f6c563114 WatchSource:0}: Error finding container 23ae41b3f5d99ad19b03c5ba5d00f6883953ffd86429c9dcd611506f6c563114: Status 404 returned error can't find the container with id 23ae41b3f5d99ad19b03c5ba5d00f6883953ffd86429c9dcd611506f6c563114
Nov 02 10:12:53.587569 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:53.587531    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k" event={"ID":"0da7b91c-3ac1-46b9-a47f-347e8775cf4b","Type":"ContainerStarted","Data":"097c34ea5ef7edec14a230b2eda6c73c78c02a4b839705a3d055603b4fb250c0"}
Nov 02 10:12:53.587759 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:53.587710    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"
Nov 02 10:12:53.587759 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:53.587743    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k" event={"ID":"0da7b91c-3ac1-46b9-a47f-347e8775cf4b","Type":"ContainerStarted","Data":"23ae41b3f5d99ad19b03c5ba5d00f6883953ffd86429c9dcd611506f6c563114"}
Nov 02 10:12:53.615538 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:53.615470    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k" podStartSLOduration=1.6154456069999998 podStartE2EDuration="1.615445607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:12:53.605307811 +0000 UTC m=+2250.543594588" watchObservedRunningTime="2026-11-02 10:12:53.615445607 +0000 UTC m=+2250.553732384"
Nov 02 10:12:59.629097 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:59.629066    4449 generic.go:356] "Generic (PLEG): container finished" podID="c00e3a95-1a6e-411e-b928-456a968e432e" containerID="f554a33b889b6ab966636a02828b65ace2029a0f21adf3bd8ffebf2008cda36a" exitCode=0
Nov 02 10:12:59.629615 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:59.629132    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq" event={"ID":"c00e3a95-1a6e-411e-b928-456a968e432e","Type":"ContainerDied","Data":"f554a33b889b6ab966636a02828b65ace2029a0f21adf3bd8ffebf2008cda36a"}
Nov 02 10:12:59.629615 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:59.629543    4449 scope.go:122] "RemoveContainer" containerID="f554a33b889b6ab966636a02828b65ace2029a0f21adf3bd8ffebf2008cda36a"
Nov 02 10:12:59.630807 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:59.630771    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:12:59.630873 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:59.630846    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:12:59.630922 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:12:59.630883    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:13:00.513942 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:00.513906    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:13:00.637600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:00.637563    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq" event={"ID":"c00e3a95-1a6e-411e-b928-456a968e432e","Type":"ContainerStarted","Data":"774677b11876f938e489c4f2a121f67a9a5c44c9dba16ca59a17a8efdc7dd2d4"}
Nov 02 10:13:00.935697 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:00.935658    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:13:00.936976 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:00.936945    4449 patch_prober.go:29] interesting pod/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 02 10:13:00.937117 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:00.936998    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq" podUID="c00e3a95-1a6e-411e-b928-456a968e432e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 02 10:13:01.936181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:01.936143    4449 patch_prober.go:29] interesting pod/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 02 10:13:01.936648 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:01.936251    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq" podUID="c00e3a95-1a6e-411e-b928-456a968e432e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 02 10:13:02.935696 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:02.935663    4449 patch_prober.go:29] interesting pod/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 02 10:13:02.935880 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:02.935720    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq" podUID="c00e3a95-1a6e-411e-b928-456a968e432e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 02 10:13:03.935984 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:03.935942    4449 patch_prober.go:29] interesting pod/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 02 10:13:03.936450 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:03.936006    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq" podUID="c00e3a95-1a6e-411e-b928-456a968e432e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 02 10:13:04.596816 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:04.596782    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"
Nov 02 10:13:04.935549 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:04.935466    4449 patch_prober.go:29] interesting pod/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 02 10:13:04.935549 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:04.935523    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq" podUID="c00e3a95-1a6e-411e-b928-456a968e432e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 02 10:13:05.940591 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:05.940557    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:13:05.941033 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:05.940845    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:13:05.941559 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:05.941544    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:13:06.547299 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:13:06.547260    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:13:13.189929 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:13.189892    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k_0da7b91c-3ac1-46b9-a47f-347e8775cf4b/payload-processing/0.log"
Nov 02 10:13:26.340850 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:26.340814    4449 scope.go:122] "RemoveContainer" containerID="0aba17c323a46ffbef5fda900fad434118ea998cda561b3866960d4fcce06ced"
Nov 02 10:13:36.546868 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:13:36.546782    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:13:48.581275 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:48.581205    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9_99ad889f-918e-4e43-b38c-b4ee2bff1825/payload-processing/0.log"
Nov 02 10:13:49.124180 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:49.124143    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k_0da7b91c-3ac1-46b9-a47f-347e8775cf4b/payload-processing/0.log"
Nov 02 10:13:53.025164 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:53.025090    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"]
Nov 02 10:13:53.028368 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:53.028345    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392f38c5-w9lff"]
Nov 02 10:13:53.206243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:13:53.206197    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9580d75-ff52-4612-b5e7-da7814cecb34" path="/var/lib/kubelet/pods/c9580d75-ff52-4612-b5e7-da7814cecb34/volumes"
Nov 02 10:14:06.547093 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:14:06.547048    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:14:10.023770 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:10.023735    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"]
Nov 02 10:14:10.026437 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:10.026414    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9f854b-7s8wl"]
Nov 02 10:14:11.205856 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:11.205823    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b84ef03b-af8f-4701-a034-744bcea14bad" path="/var/lib/kubelet/pods/b84ef03b-af8f-4701-a034-744bcea14bad/volumes"
Nov 02 10:14:26.369794 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:26.369762    4449 scope.go:122] "RemoveContainer" containerID="98cc4ea2ef7c7248b8c71153f6ade6deb55ecff0b647ef98f52d39a07132e4f9"
Nov 02 10:14:26.379428 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:26.379402    4449 scope.go:122] "RemoveContainer" containerID="bcfa43581b7437eed89e524aceba4be5d03914078760f2f1a9abb5a089361b30"
Nov 02 10:14:26.388501 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:26.388480    4449 scope.go:122] "RemoveContainer" containerID="f93d686c6bb3104163ab5dcabaa52cf7c64521e0d7913c3dc5c135be298587fd"
Nov 02 10:14:36.547597 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:14:36.547560    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:14:48.085892 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:48.085851    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"]
Nov 02 10:14:48.086481 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:48.086127    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9" podUID="99ad889f-918e-4e43-b38c-b4ee2bff1825" containerName="payload-processing" containerID="cri-o://b27b274203105cb4ba70650b1fa9c1aa3234a577e6d4b66246f2b13cf17fc814" gracePeriod=30
Nov 02 10:14:48.355272 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:48.355234    4449 generic.go:356] "Generic (PLEG): container finished" podID="99ad889f-918e-4e43-b38c-b4ee2bff1825" containerID="b27b274203105cb4ba70650b1fa9c1aa3234a577e6d4b66246f2b13cf17fc814" exitCode=0
Nov 02 10:14:48.355460 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:48.355435    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9" event={"ID":"99ad889f-918e-4e43-b38c-b4ee2bff1825","Type":"ContainerDied","Data":"b27b274203105cb4ba70650b1fa9c1aa3234a577e6d4b66246f2b13cf17fc814"}
Nov 02 10:14:48.376778 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:48.376752    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"
Nov 02 10:14:48.557068 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:48.557032    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ad889f-918e-4e43-b38c-b4ee2bff1825-kube-api-access-7lj6t\" (UniqueName: \"kubernetes.io/projected/99ad889f-918e-4e43-b38c-b4ee2bff1825-kube-api-access-7lj6t\") pod \"99ad889f-918e-4e43-b38c-b4ee2bff1825\" (UID: \"99ad889f-918e-4e43-b38c-b4ee2bff1825\") "
Nov 02 10:14:48.557322 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:48.557304    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ad889f-918e-4e43-b38c-b4ee2bff1825-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ad889f-918e-4e43-b38c-b4ee2bff1825-plugins-config-volume\") pod \"99ad889f-918e-4e43-b38c-b4ee2bff1825\" (UID: \"99ad889f-918e-4e43-b38c-b4ee2bff1825\") "
Nov 02 10:14:48.557710 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:48.557682    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ad889f-918e-4e43-b38c-b4ee2bff1825-plugins-config-volume" pod "99ad889f-918e-4e43-b38c-b4ee2bff1825" (UID: "99ad889f-918e-4e43-b38c-b4ee2bff1825"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:14:48.559709 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:48.559675    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ad889f-918e-4e43-b38c-b4ee2bff1825-kube-api-access-7lj6t" pod "99ad889f-918e-4e43-b38c-b4ee2bff1825" (UID: "99ad889f-918e-4e43-b38c-b4ee2bff1825"). InnerVolumeSpecName "kube-api-access-7lj6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:14:48.658420 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:48.658334    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ad889f-918e-4e43-b38c-b4ee2bff1825-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:48.658420 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:48.658366    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lj6t\" (UniqueName: \"kubernetes.io/projected/99ad889f-918e-4e43-b38c-b4ee2bff1825-kube-api-access-7lj6t\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:49.362555 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:49.362525    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"
Nov 02 10:14:49.362962 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:49.362552    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9" event={"ID":"99ad889f-918e-4e43-b38c-b4ee2bff1825","Type":"ContainerDied","Data":"de13883aba7ead8498c181dd2769d3ce314df4b4ee75c9f6949e34289d4fe053"}
Nov 02 10:14:49.362962 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:49.362621    4449 scope.go:122] "RemoveContainer" containerID="b27b274203105cb4ba70650b1fa9c1aa3234a577e6d4b66246f2b13cf17fc814"
Nov 02 10:14:49.381459 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:49.381429    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"]
Nov 02 10:14:49.385714 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:49.385687    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2b5a18-76496b8844-xjpx9"]
Nov 02 10:14:51.206708 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:51.206672    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ad889f-918e-4e43-b38c-b4ee2bff1825" path="/var/lib/kubelet/pods/99ad889f-918e-4e43-b38c-b4ee2bff1825/volumes"
Nov 02 10:14:53.930547 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:53.930496    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"]
Nov 02 10:14:53.938321 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:53.938276    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerName="istio-proxy" containerID="cri-o://68120099b422d73721e41109aa2531aea5bd962c484d338b3778e382e090983d" gracePeriod=30
Nov 02 10:14:59.212804 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.212776    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:14:59.273990 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.273900    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-token\" (UniqueName: \"kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-token\") pod \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") "
Nov 02 10:14:59.273990 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.273982    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-data\") pod \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") "
Nov 02 10:14:59.274263 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.274038    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-podinfo\") pod \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") "
Nov 02 10:14:59.274263 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.274079    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-kube-api-access-nlhzx\" (UniqueName: \"kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-kube-api-access-nlhzx\") pod \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") "
Nov 02 10:14:59.274263 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.274126    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istiod-ca-cert\") pod \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") "
Nov 02 10:14:59.274263 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.274151    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-credential-socket\") pod \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") "
Nov 02 10:14:59.274263 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.274211    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-certs\") pod \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") "
Nov 02 10:14:59.274531 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.274283    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-socket\") pod \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") "
Nov 02 10:14:59.274531 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.274295    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-data" pod "abcc4858-2830-479c-9e88-1bde7a9e1a0a" (UID: "abcc4858-2830-479c-9e88-1bde7a9e1a0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:14:59.274531 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.274363    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-envoy\") pod \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\" (UID: \"abcc4858-2830-479c-9e88-1bde7a9e1a0a\") "
Nov 02 10:14:59.274690 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.274600    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-certs" pod "abcc4858-2830-479c-9e88-1bde7a9e1a0a" (UID: "abcc4858-2830-479c-9e88-1bde7a9e1a0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:14:59.274690 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.274626    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istiod-ca-cert" pod "abcc4858-2830-479c-9e88-1bde7a9e1a0a" (UID: "abcc4858-2830-479c-9e88-1bde7a9e1a0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:14:59.274801 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.274782    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-credential-socket" pod "abcc4858-2830-479c-9e88-1bde7a9e1a0a" (UID: "abcc4858-2830-479c-9e88-1bde7a9e1a0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:14:59.274862 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.274815    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-socket" pod "abcc4858-2830-479c-9e88-1bde7a9e1a0a" (UID: "abcc4858-2830-479c-9e88-1bde7a9e1a0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:14:59.275367 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.275271    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:59.275367 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.275295    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:59.275367 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.275310    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:59.275367 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.275326    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:59.275367 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.275340    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:59.277480 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.277449    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-kube-api-access-nlhzx" pod "abcc4858-2830-479c-9e88-1bde7a9e1a0a" (UID: "abcc4858-2830-479c-9e88-1bde7a9e1a0a"). InnerVolumeSpecName "kube-api-access-nlhzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:14:59.277600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.277512    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-podinfo" pod "abcc4858-2830-479c-9e88-1bde7a9e1a0a" (UID: "abcc4858-2830-479c-9e88-1bde7a9e1a0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:14:59.277600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.277526    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-envoy" pod "abcc4858-2830-479c-9e88-1bde7a9e1a0a" (UID: "abcc4858-2830-479c-9e88-1bde7a9e1a0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:14:59.277600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.277585    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-token" pod "abcc4858-2830-479c-9e88-1bde7a9e1a0a" (UID: "abcc4858-2830-479c-9e88-1bde7a9e1a0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:14:59.376482 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.376426    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:59.376482 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.376477    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:59.376482 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.376488    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlhzx\" (UniqueName: \"kubernetes.io/projected/abcc4858-2830-479c-9e88-1bde7a9e1a0a-kube-api-access-nlhzx\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:59.376737 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.376498    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abcc4858-2830-479c-9e88-1bde7a9e1a0a-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:59.429430 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.429398    4449 generic.go:356] "Generic (PLEG): container finished" podID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerID="68120099b422d73721e41109aa2531aea5bd962c484d338b3778e382e090983d" exitCode=0
Nov 02 10:14:59.429639 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.429440    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" event={"ID":"abcc4858-2830-479c-9e88-1bde7a9e1a0a","Type":"ContainerDied","Data":"68120099b422d73721e41109aa2531aea5bd962c484d338b3778e382e090983d"}
Nov 02 10:14:59.429639 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.429462    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"
Nov 02 10:14:59.429639 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.429492    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9" event={"ID":"abcc4858-2830-479c-9e88-1bde7a9e1a0a","Type":"ContainerDied","Data":"29fb5f6dd2cf38c9e4323a7cfa8c2fb55b23a12ce94c55c38b34e1f55c3b1072"}
Nov 02 10:14:59.429639 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.429520    4449 scope.go:122] "RemoveContainer" containerID="68120099b422d73721e41109aa2531aea5bd962c484d338b3778e382e090983d"
Nov 02 10:14:59.439908 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.439890    4449 scope.go:122] "RemoveContainer" containerID="d4a153e9101c74d93a4466fd0c2cc66ad99016d7c45dcfef0419000ab66e578a"
Nov 02 10:14:59.449815 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.449796    4449 scope.go:122] "RemoveContainer" containerID="68120099b422d73721e41109aa2531aea5bd962c484d338b3778e382e090983d"
Nov 02 10:14:59.450110 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:14:59.450082    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68120099b422d73721e41109aa2531aea5bd962c484d338b3778e382e090983d\": container with ID starting with 68120099b422d73721e41109aa2531aea5bd962c484d338b3778e382e090983d not found: ID does not exist" containerID="68120099b422d73721e41109aa2531aea5bd962c484d338b3778e382e090983d"
Nov 02 10:14:59.450180 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.450123    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68120099b422d73721e41109aa2531aea5bd962c484d338b3778e382e090983d"} err="failed to get container status \"68120099b422d73721e41109aa2531aea5bd962c484d338b3778e382e090983d\": rpc error: code = NotFound desc = could not find container \"68120099b422d73721e41109aa2531aea5bd962c484d338b3778e382e090983d\": container with ID starting with 68120099b422d73721e41109aa2531aea5bd962c484d338b3778e382e090983d not found: ID does not exist"
Nov 02 10:14:59.450180 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.450145    4449 scope.go:122] "RemoveContainer" containerID="d4a153e9101c74d93a4466fd0c2cc66ad99016d7c45dcfef0419000ab66e578a"
Nov 02 10:14:59.450488 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:14:59.450466    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a153e9101c74d93a4466fd0c2cc66ad99016d7c45dcfef0419000ab66e578a\": container with ID starting with d4a153e9101c74d93a4466fd0c2cc66ad99016d7c45dcfef0419000ab66e578a not found: ID does not exist" containerID="d4a153e9101c74d93a4466fd0c2cc66ad99016d7c45dcfef0419000ab66e578a"
Nov 02 10:14:59.450545 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.450504    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a153e9101c74d93a4466fd0c2cc66ad99016d7c45dcfef0419000ab66e578a"} err="failed to get container status \"d4a153e9101c74d93a4466fd0c2cc66ad99016d7c45dcfef0419000ab66e578a\": rpc error: code = NotFound desc = could not find container \"d4a153e9101c74d93a4466fd0c2cc66ad99016d7c45dcfef0419000ab66e578a\": container with ID starting with d4a153e9101c74d93a4466fd0c2cc66ad99016d7c45dcfef0419000ab66e578a not found: ID does not exist"
Nov 02 10:14:59.455496 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.455469    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"]
Nov 02 10:14:59.458618 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:14:59.458589    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2b5a18-openshift-default-7d44b947f7-5klp9"]
Nov 02 10:15:00.139179 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.139143    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"]
Nov 02 10:15:00.139644 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.139630    4449 cpu_manager.go:414] "containerMap: removing container" podUID="99ad889f-918e-4e43-b38c-b4ee2bff1825" containerName="payload-processing"
Nov 02 10:15:00.139696 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.139646    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ad889f-918e-4e43-b38c-b4ee2bff1825" containerName="payload-processing"
Nov 02 10:15:00.139696 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.139680    4449 cpu_manager.go:414] "containerMap: removing container" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerName="istio-proxy"
Nov 02 10:15:00.139696 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.139686    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerName="istio-proxy"
Nov 02 10:15:00.139831 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.139699    4449 cpu_manager.go:414] "containerMap: removing container" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerName="istio-proxy"
Nov 02 10:15:00.139831 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.139704    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerName="istio-proxy"
Nov 02 10:15:00.139831 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.139767    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerName="istio-proxy"
Nov 02 10:15:00.139831 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.139777    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" containerName="istio-proxy"
Nov 02 10:15:00.139831 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.139786    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ad889f-918e-4e43-b38c-b4ee2bff1825" containerName="payload-processing"
Nov 02 10:15:00.144383 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.144366    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"
Nov 02 10:15:00.151765 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.151741    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:15:00.156392 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.153030    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:15:00.156392 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.153112    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:15:00.156392 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.155772    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"]
Nov 02 10:15:00.185656 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.185613    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nvbg\" (UniqueName: \"kubernetes.io/projected/0c5e3e67-508a-4d81-851b-656fa138e61d-kube-api-access-5nvbg\") pod \"maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h\" (UID: \"0c5e3e67-508a-4d81-851b-656fa138e61d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"
Nov 02 10:15:00.185880 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.185848    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5e3e67-508a-4d81-851b-656fa138e61d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h\" (UID: \"0c5e3e67-508a-4d81-851b-656fa138e61d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"
Nov 02 10:15:00.287246 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.287177    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5e3e67-508a-4d81-851b-656fa138e61d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h\" (UID: \"0c5e3e67-508a-4d81-851b-656fa138e61d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"
Nov 02 10:15:00.287738 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.287304    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nvbg\" (UniqueName: \"kubernetes.io/projected/0c5e3e67-508a-4d81-851b-656fa138e61d-kube-api-access-5nvbg\") pod \"maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h\" (UID: \"0c5e3e67-508a-4d81-851b-656fa138e61d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"
Nov 02 10:15:00.287943 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.287919    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5e3e67-508a-4d81-851b-656fa138e61d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h\" (UID: \"0c5e3e67-508a-4d81-851b-656fa138e61d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"
Nov 02 10:15:00.299821 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.299793    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nvbg\" (UniqueName: \"kubernetes.io/projected/0c5e3e67-508a-4d81-851b-656fa138e61d-kube-api-access-5nvbg\") pod \"maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h\" (UID: \"0c5e3e67-508a-4d81-851b-656fa138e61d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"
Nov 02 10:15:00.460636 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.460539    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"
Nov 02 10:15:00.739967 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:00.739941    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"]
Nov 02 10:15:00.742430 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:15:00.742402    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5e3e67_508a_4d81_851b_656fa138e61d.slice/crio-8877fb05df48a1c843a63cac17fbcffa5a7ad7de84e1e553c96c23bbc9d420cf WatchSource:0}: Error finding container 8877fb05df48a1c843a63cac17fbcffa5a7ad7de84e1e553c96c23bbc9d420cf: Status 404 returned error can't find the container with id 8877fb05df48a1c843a63cac17fbcffa5a7ad7de84e1e553c96c23bbc9d420cf
Nov 02 10:15:01.205007 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:01.204975    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abcc4858-2830-479c-9e88-1bde7a9e1a0a" path="/var/lib/kubelet/pods/abcc4858-2830-479c-9e88-1bde7a9e1a0a/volumes"
Nov 02 10:15:01.444447 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:01.444357    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h" event={"ID":"0c5e3e67-508a-4d81-851b-656fa138e61d","Type":"ContainerStarted","Data":"59b57ad09ca79693c3371a5bd55975b6f9e4a30176ef7e3346ff019ff5631ce5"}
Nov 02 10:15:01.444447 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:01.444416    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h" event={"ID":"0c5e3e67-508a-4d81-851b-656fa138e61d","Type":"ContainerStarted","Data":"8877fb05df48a1c843a63cac17fbcffa5a7ad7de84e1e553c96c23bbc9d420cf"}
Nov 02 10:15:01.472825 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:01.472765    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h" podStartSLOduration=1.472745566 podStartE2EDuration="1.472745566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:15:01.46732185 +0000 UTC m=+2378.405608646" watchObservedRunningTime="2026-11-02 10:15:01.472745566 +0000 UTC m=+2378.411032326"
Nov 02 10:15:02.451701 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:02.451668    4449 generic.go:356] "Generic (PLEG): container finished" podID="0c5e3e67-508a-4d81-851b-656fa138e61d" containerID="59b57ad09ca79693c3371a5bd55975b6f9e4a30176ef7e3346ff019ff5631ce5" exitCode=60
Nov 02 10:15:02.452177 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:02.451753    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h" event={"ID":"0c5e3e67-508a-4d81-851b-656fa138e61d","Type":"ContainerDied","Data":"59b57ad09ca79693c3371a5bd55975b6f9e4a30176ef7e3346ff019ff5631ce5"}
Nov 02 10:15:02.452177 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:02.452057    4449 scope.go:122] "RemoveContainer" containerID="59b57ad09ca79693c3371a5bd55975b6f9e4a30176ef7e3346ff019ff5631ce5"
Nov 02 10:15:03.460489 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:03.460440    4449 generic.go:356] "Generic (PLEG): container finished" podID="0c5e3e67-508a-4d81-851b-656fa138e61d" containerID="7bbce12a4fa1e83e43872d0e1f1f791fb68e1a07393b512a8732cd4b45932a40" exitCode=60
Nov 02 10:15:03.460908 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:03.460527    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h" event={"ID":"0c5e3e67-508a-4d81-851b-656fa138e61d","Type":"ContainerDied","Data":"7bbce12a4fa1e83e43872d0e1f1f791fb68e1a07393b512a8732cd4b45932a40"}
Nov 02 10:15:03.460908 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:03.460611    4449 scope.go:122] "RemoveContainer" containerID="59b57ad09ca79693c3371a5bd55975b6f9e4a30176ef7e3346ff019ff5631ce5"
Nov 02 10:15:03.461066 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:03.460933    4449 scope.go:122] "RemoveContainer" containerID="7bbce12a4fa1e83e43872d0e1f1f791fb68e1a07393b512a8732cd4b45932a40"
Nov 02 10:15:03.461274 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:15:03.461254    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h_odh-ai-gateway-infra(0c5e3e67-508a-4d81-851b-656fa138e61d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h" podUID="0c5e3e67-508a-4d81-851b-656fa138e61d"
Nov 02 10:15:04.479163 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:04.479089    4449 scope.go:122] "RemoveContainer" containerID="7bbce12a4fa1e83e43872d0e1f1f791fb68e1a07393b512a8732cd4b45932a40"
Nov 02 10:15:04.479554 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:15:04.479392    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h_odh-ai-gateway-infra(0c5e3e67-508a-4d81-851b-656fa138e61d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h" podUID="0c5e3e67-508a-4d81-851b-656fa138e61d"
Nov 02 10:15:06.546503 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:15:06.546465    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:15:07.315690 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.315654    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"]
Nov 02 10:15:07.369174 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.369084    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"]
Nov 02 10:15:07.369432 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.369383    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k" podUID="0da7b91c-3ac1-46b9-a47f-347e8775cf4b" containerName="payload-processing" containerID="cri-o://097c34ea5ef7edec14a230b2eda6c73c78c02a4b839705a3d055603b4fb250c0" gracePeriod=30
Nov 02 10:15:07.498905 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.498872    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h" event={"ID":"0c5e3e67-508a-4d81-851b-656fa138e61d","Type":"ContainerDied","Data":"8877fb05df48a1c843a63cac17fbcffa5a7ad7de84e1e553c96c23bbc9d420cf"}
Nov 02 10:15:07.499044 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.498934    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8877fb05df48a1c843a63cac17fbcffa5a7ad7de84e1e553c96c23bbc9d420cf"
Nov 02 10:15:07.501916 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.501867    4449 generic.go:356] "Generic (PLEG): container finished" podID="0da7b91c-3ac1-46b9-a47f-347e8775cf4b" containerID="097c34ea5ef7edec14a230b2eda6c73c78c02a4b839705a3d055603b4fb250c0" exitCode=0
Nov 02 10:15:07.502036 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.501935    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k" event={"ID":"0da7b91c-3ac1-46b9-a47f-347e8775cf4b","Type":"ContainerDied","Data":"097c34ea5ef7edec14a230b2eda6c73c78c02a4b839705a3d055603b4fb250c0"}
Nov 02 10:15:07.503003 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.502989    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"
Nov 02 10:15:07.576292 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.575449    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c5e3e67-508a-4d81-851b-656fa138e61d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5e3e67-508a-4d81-851b-656fa138e61d-maas-api-service-ca\") pod \"0c5e3e67-508a-4d81-851b-656fa138e61d\" (UID: \"0c5e3e67-508a-4d81-851b-656fa138e61d\") "
Nov 02 10:15:07.576292 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.575597    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5e3e67-508a-4d81-851b-656fa138e61d-kube-api-access-5nvbg\" (UniqueName: \"kubernetes.io/projected/0c5e3e67-508a-4d81-851b-656fa138e61d-kube-api-access-5nvbg\") pod \"0c5e3e67-508a-4d81-851b-656fa138e61d\" (UID: \"0c5e3e67-508a-4d81-851b-656fa138e61d\") "
Nov 02 10:15:07.594189 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.587645    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c5e3e67-508a-4d81-851b-656fa138e61d-maas-api-service-ca" pod "0c5e3e67-508a-4d81-851b-656fa138e61d" (UID: "0c5e3e67-508a-4d81-851b-656fa138e61d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:15:07.598820 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.598776    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5e3e67-508a-4d81-851b-656fa138e61d-kube-api-access-5nvbg" pod "0c5e3e67-508a-4d81-851b-656fa138e61d" (UID: "0c5e3e67-508a-4d81-851b-656fa138e61d"). InnerVolumeSpecName "kube-api-access-5nvbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:15:07.659646 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.659615    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"
Nov 02 10:15:07.676823 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.676784    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5e3e67-508a-4d81-851b-656fa138e61d-maas-api-service-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:07.676823 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.676813    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nvbg\" (UniqueName: \"kubernetes.io/projected/0c5e3e67-508a-4d81-851b-656fa138e61d-kube-api-access-5nvbg\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:07.777469 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.777422    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-kube-api-access-4rsjn\" (UniqueName: \"kubernetes.io/projected/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-kube-api-access-4rsjn\") pod \"0da7b91c-3ac1-46b9-a47f-347e8775cf4b\" (UID: \"0da7b91c-3ac1-46b9-a47f-347e8775cf4b\") "
Nov 02 10:15:07.777692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.777518    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-plugins-config-volume\") pod \"0da7b91c-3ac1-46b9-a47f-347e8775cf4b\" (UID: \"0da7b91c-3ac1-46b9-a47f-347e8775cf4b\") "
Nov 02 10:15:07.777994 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.777965    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-plugins-config-volume" pod "0da7b91c-3ac1-46b9-a47f-347e8775cf4b" (UID: "0da7b91c-3ac1-46b9-a47f-347e8775cf4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:15:07.779679 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.779654    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-kube-api-access-4rsjn" pod "0da7b91c-3ac1-46b9-a47f-347e8775cf4b" (UID: "0da7b91c-3ac1-46b9-a47f-347e8775cf4b"). InnerVolumeSpecName "kube-api-access-4rsjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:15:07.879621 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.879536    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rsjn\" (UniqueName: \"kubernetes.io/projected/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-kube-api-access-4rsjn\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:07.879621 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:07.879563    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da7b91c-3ac1-46b9-a47f-347e8775cf4b-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:08.508937 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:08.508910    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"
Nov 02 10:15:08.509155 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:08.508945    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k" event={"ID":"0da7b91c-3ac1-46b9-a47f-347e8775cf4b","Type":"ContainerDied","Data":"23ae41b3f5d99ad19b03c5ba5d00f6883953ffd86429c9dcd611506f6c563114"}
Nov 02 10:15:08.509155 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:08.509000    4449 scope.go:122] "RemoveContainer" containerID="097c34ea5ef7edec14a230b2eda6c73c78c02a4b839705a3d055603b4fb250c0"
Nov 02 10:15:08.509321 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:08.509241    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"
Nov 02 10:15:08.540728 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:08.540702    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"]
Nov 02 10:15:08.544941 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:08.544912    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-836bba-7b8f49cd68-2q26k"]
Nov 02 10:15:08.556969 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:08.556935    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"]
Nov 02 10:15:08.561143 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:08.561113    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-836bba-29893575-6nt8h"]
Nov 02 10:15:09.024140 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:09.024085    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"]
Nov 02 10:15:09.026557 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:09.026534    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-651fee84-ldqmj"]
Nov 02 10:15:09.205534 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:09.205503    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c5e3e67-508a-4d81-851b-656fa138e61d" path="/var/lib/kubelet/pods/0c5e3e67-508a-4d81-851b-656fa138e61d/volumes"
Nov 02 10:15:09.205896 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:09.205883    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da7b91c-3ac1-46b9-a47f-347e8775cf4b" path="/var/lib/kubelet/pods/0da7b91c-3ac1-46b9-a47f-347e8775cf4b/volumes"
Nov 02 10:15:09.206289 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:09.206275    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7030264-16be-48b6-ba8d-d526df941d82" path="/var/lib/kubelet/pods/a7030264-16be-48b6-ba8d-d526df941d82/volumes"
Nov 02 10:15:13.340207 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:13.340174    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"]
Nov 02 10:15:13.340744 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:13.340445    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq" podUID="c00e3a95-1a6e-411e-b928-456a968e432e" containerName="istio-proxy" containerID="cri-o://774677b11876f938e489c4f2a121f67a9a5c44c9dba16ca59a17a8efdc7dd2d4" gracePeriod=30
Nov 02 10:15:18.570352 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.570321    4449 generic.go:356] "Generic (PLEG): container finished" podID="c00e3a95-1a6e-411e-b928-456a968e432e" containerID="774677b11876f938e489c4f2a121f67a9a5c44c9dba16ca59a17a8efdc7dd2d4" exitCode=0
Nov 02 10:15:18.570778 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.570395    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq" event={"ID":"c00e3a95-1a6e-411e-b928-456a968e432e","Type":"ContainerDied","Data":"774677b11876f938e489c4f2a121f67a9a5c44c9dba16ca59a17a8efdc7dd2d4"}
Nov 02 10:15:18.570778 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.570457    4449 scope.go:122] "RemoveContainer" containerID="f554a33b889b6ab966636a02828b65ace2029a0f21adf3bd8ffebf2008cda36a"
Nov 02 10:15:18.607294 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.607234    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:15:18.795197 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.795161    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-envoy\") pod \"c00e3a95-1a6e-411e-b928-456a968e432e\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") "
Nov 02 10:15:18.795421 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.795209    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00e3a95-1a6e-411e-b928-456a968e432e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00e3a95-1a6e-411e-b928-456a968e432e-istiod-ca-cert\") pod \"c00e3a95-1a6e-411e-b928-456a968e432e\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") "
Nov 02 10:15:18.795421 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.795273    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-certs\") pod \"c00e3a95-1a6e-411e-b928-456a968e432e\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") "
Nov 02 10:15:18.795421 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.795334    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-data\") pod \"c00e3a95-1a6e-411e-b928-456a968e432e\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") "
Nov 02 10:15:18.795421 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.795358    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-kube-api-access-svnkd\" (UniqueName: \"kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-kube-api-access-svnkd\") pod \"c00e3a95-1a6e-411e-b928-456a968e432e\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") "
Nov 02 10:15:18.795421 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.795376    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-istio-token\" (UniqueName: \"kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-istio-token\") pod \"c00e3a95-1a6e-411e-b928-456a968e432e\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") "
Nov 02 10:15:18.795421 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.795405    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-credential-socket\") pod \"c00e3a95-1a6e-411e-b928-456a968e432e\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") "
Nov 02 10:15:18.795751 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.795440    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-socket\") pod \"c00e3a95-1a6e-411e-b928-456a968e432e\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") "
Nov 02 10:15:18.795751 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.795544    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c00e3a95-1a6e-411e-b928-456a968e432e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00e3a95-1a6e-411e-b928-456a968e432e-istio-podinfo\") pod \"c00e3a95-1a6e-411e-b928-456a968e432e\" (UID: \"c00e3a95-1a6e-411e-b928-456a968e432e\") "
Nov 02 10:15:18.795751 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.795642    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00e3a95-1a6e-411e-b928-456a968e432e-istiod-ca-cert" pod "c00e3a95-1a6e-411e-b928-456a968e432e" (UID: "c00e3a95-1a6e-411e-b928-456a968e432e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:15:18.795751 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.795655    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-data" pod "c00e3a95-1a6e-411e-b928-456a968e432e" (UID: "c00e3a95-1a6e-411e-b928-456a968e432e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:15:18.795961 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.795882    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-credential-socket" pod "c00e3a95-1a6e-411e-b928-456a968e432e" (UID: "c00e3a95-1a6e-411e-b928-456a968e432e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:15:18.795961 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.795895    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-socket" pod "c00e3a95-1a6e-411e-b928-456a968e432e" (UID: "c00e3a95-1a6e-411e-b928-456a968e432e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:15:18.796076 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.796007    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:18.796076 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.796029    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:18.796076 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.796046    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:18.796076 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.796049    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-certs" pod "c00e3a95-1a6e-411e-b928-456a968e432e" (UID: "c00e3a95-1a6e-411e-b928-456a968e432e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:15:18.796277 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.796059    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00e3a95-1a6e-411e-b928-456a968e432e-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:18.797504 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.797480    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-envoy" pod "c00e3a95-1a6e-411e-b928-456a968e432e" (UID: "c00e3a95-1a6e-411e-b928-456a968e432e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:15:18.797932 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.797916    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-istio-token" pod "c00e3a95-1a6e-411e-b928-456a968e432e" (UID: "c00e3a95-1a6e-411e-b928-456a968e432e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:15:18.798382 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.798369    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-kube-api-access-svnkd" pod "c00e3a95-1a6e-411e-b928-456a968e432e" (UID: "c00e3a95-1a6e-411e-b928-456a968e432e"). InnerVolumeSpecName "kube-api-access-svnkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:15:18.798437 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.798396    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c00e3a95-1a6e-411e-b928-456a968e432e-istio-podinfo" pod "c00e3a95-1a6e-411e-b928-456a968e432e" (UID: "c00e3a95-1a6e-411e-b928-456a968e432e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:15:18.896756 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.896670    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:18.896756 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.896701    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00e3a95-1a6e-411e-b928-456a968e432e-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:18.896756 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.896712    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svnkd\" (UniqueName: \"kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-kube-api-access-svnkd\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:18.896756 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.896722    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c00e3a95-1a6e-411e-b928-456a968e432e-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:18.896756 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:18.896730    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00e3a95-1a6e-411e-b928-456a968e432e-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:19.577187 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:19.577157    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"
Nov 02 10:15:19.577734 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:19.577199    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq" event={"ID":"c00e3a95-1a6e-411e-b928-456a968e432e","Type":"ContainerDied","Data":"98f142998cf93cac2b29a0e00e3a3342c7a69fb16eb3186d7fe9e47bfc85d95e"}
Nov 02 10:15:19.577734 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:19.577279    4449 scope.go:122] "RemoveContainer" containerID="774677b11876f938e489c4f2a121f67a9a5c44c9dba16ca59a17a8efdc7dd2d4"
Nov 02 10:15:19.598612 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:19.598579    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"]
Nov 02 10:15:19.611204 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:19.611170    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-836bba-openshift-default-887b755d9-l5rmq"]
Nov 02 10:15:21.205496 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:21.205466    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00e3a95-1a6e-411e-b928-456a968e432e" path="/var/lib/kubelet/pods/c00e3a95-1a6e-411e-b928-456a968e432e/volumes"
Nov 02 10:15:26.421898 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:26.421865    4449 scope.go:122] "RemoveContainer" containerID="83d583919c2db78649d46c94f5fdb668a6308c740dcdd0bba6eaf670ac08b93a"
Nov 02 10:15:26.431023 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:26.431000    4449 scope.go:122] "RemoveContainer" containerID="a6fbf15a35e9070b766964ffdb5cfbe2f2e311b2bdc1189a78a42fdb90a13fec"
Nov 02 10:15:27.655062 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:27.655030    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"]
Nov 02 10:15:32.655964 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:32.655893    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9" podUID="3c434dd2-b08d-4ecc-9ab2-d89ab694963b" containerName="payload-pre-processing" containerID="cri-o://616f43151913965db2fa4a00792856d1a3e2d98f19415d70f5d32bc4e39cde67" gracePeriod=25
Nov 02 10:15:32.918208 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:32.918140    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"
Nov 02 10:15:33.043980 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.043951    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-kube-api-access-nn4wp\" (UniqueName: \"kubernetes.io/projected/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-kube-api-access-nn4wp\") pod \"3c434dd2-b08d-4ecc-9ab2-d89ab694963b\" (UID: \"3c434dd2-b08d-4ecc-9ab2-d89ab694963b\") "
Nov 02 10:15:33.044376 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.044347    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-plugins-config-volume\") pod \"3c434dd2-b08d-4ecc-9ab2-d89ab694963b\" (UID: \"3c434dd2-b08d-4ecc-9ab2-d89ab694963b\") "
Nov 02 10:15:33.045104 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.044987    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-plugins-config-volume" pod "3c434dd2-b08d-4ecc-9ab2-d89ab694963b" (UID: "3c434dd2-b08d-4ecc-9ab2-d89ab694963b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:15:33.046867 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.046837    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-kube-api-access-nn4wp" pod "3c434dd2-b08d-4ecc-9ab2-d89ab694963b" (UID: "3c434dd2-b08d-4ecc-9ab2-d89ab694963b"). InnerVolumeSpecName "kube-api-access-nn4wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:15:33.145739 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.145692    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn4wp\" (UniqueName: \"kubernetes.io/projected/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-kube-api-access-nn4wp\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:33.145739 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.145728    4449 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c434dd2-b08d-4ecc-9ab2-d89ab694963b-plugins-config-volume\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:33.659633 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.659594    4449 generic.go:356] "Generic (PLEG): container finished" podID="3c434dd2-b08d-4ecc-9ab2-d89ab694963b" containerID="616f43151913965db2fa4a00792856d1a3e2d98f19415d70f5d32bc4e39cde67" exitCode=0
Nov 02 10:15:33.659633 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.659634    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"
Nov 02 10:15:33.660254 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.659679    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9" event={"ID":"3c434dd2-b08d-4ecc-9ab2-d89ab694963b","Type":"ContainerDied","Data":"616f43151913965db2fa4a00792856d1a3e2d98f19415d70f5d32bc4e39cde67"}
Nov 02 10:15:33.660254 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.659730    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9" event={"ID":"3c434dd2-b08d-4ecc-9ab2-d89ab694963b","Type":"ContainerDied","Data":"833b6a6d87f5fb42e79f719037e887f7ab210fe529ebea995f8b5963d9acc500"}
Nov 02 10:15:33.660254 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.659756    4449 scope.go:122] "RemoveContainer" containerID="616f43151913965db2fa4a00792856d1a3e2d98f19415d70f5d32bc4e39cde67"
Nov 02 10:15:33.674191 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.674168    4449 scope.go:122] "RemoveContainer" containerID="616f43151913965db2fa4a00792856d1a3e2d98f19415d70f5d32bc4e39cde67"
Nov 02 10:15:33.674534 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:15:33.674513    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616f43151913965db2fa4a00792856d1a3e2d98f19415d70f5d32bc4e39cde67\": container with ID starting with 616f43151913965db2fa4a00792856d1a3e2d98f19415d70f5d32bc4e39cde67 not found: ID does not exist" containerID="616f43151913965db2fa4a00792856d1a3e2d98f19415d70f5d32bc4e39cde67"
Nov 02 10:15:33.674620 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.674547    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616f43151913965db2fa4a00792856d1a3e2d98f19415d70f5d32bc4e39cde67"} err="failed to get container status \"616f43151913965db2fa4a00792856d1a3e2d98f19415d70f5d32bc4e39cde67\": rpc error: code = NotFound desc = could not find container \"616f43151913965db2fa4a00792856d1a3e2d98f19415d70f5d32bc4e39cde67\": container with ID starting with 616f43151913965db2fa4a00792856d1a3e2d98f19415d70f5d32bc4e39cde67 not found: ID does not exist"
Nov 02 10:15:33.681500 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.681474    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"]
Nov 02 10:15:33.685749 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:33.685724    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j6sq9"]
Nov 02 10:15:35.204637 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:35.204594    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c434dd2-b08d-4ecc-9ab2-d89ab694963b" path="/var/lib/kubelet/pods/3c434dd2-b08d-4ecc-9ab2-d89ab694963b/volumes"
Nov 02 10:15:36.546349 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:15:36.546311    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:15:43.681930 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.681892    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-75fb58b5b7-xbsfb"]
Nov 02 10:15:43.682414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682338    4449 cpu_manager.go:414] "containerMap: removing container" podUID="c00e3a95-1a6e-411e-b928-456a968e432e" containerName="istio-proxy"
Nov 02 10:15:43.682414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682351    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00e3a95-1a6e-411e-b928-456a968e432e" containerName="istio-proxy"
Nov 02 10:15:43.682414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682366    4449 cpu_manager.go:414] "containerMap: removing container" podUID="3c434dd2-b08d-4ecc-9ab2-d89ab694963b" containerName="payload-pre-processing"
Nov 02 10:15:43.682414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682371    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c434dd2-b08d-4ecc-9ab2-d89ab694963b" containerName="payload-pre-processing"
Nov 02 10:15:43.682414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682380    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0c5e3e67-508a-4d81-851b-656fa138e61d" containerName="cleanup"
Nov 02 10:15:43.682414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682384    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5e3e67-508a-4d81-851b-656fa138e61d" containerName="cleanup"
Nov 02 10:15:43.682414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682392    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0da7b91c-3ac1-46b9-a47f-347e8775cf4b" containerName="payload-processing"
Nov 02 10:15:43.682414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682396    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da7b91c-3ac1-46b9-a47f-347e8775cf4b" containerName="payload-processing"
Nov 02 10:15:43.682414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682411    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0c5e3e67-508a-4d81-851b-656fa138e61d" containerName="cleanup"
Nov 02 10:15:43.682414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682417    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5e3e67-508a-4d81-851b-656fa138e61d" containerName="cleanup"
Nov 02 10:15:43.682755 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682503    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00e3a95-1a6e-411e-b928-456a968e432e" containerName="istio-proxy"
Nov 02 10:15:43.682755 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682511    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c434dd2-b08d-4ecc-9ab2-d89ab694963b" containerName="payload-pre-processing"
Nov 02 10:15:43.682755 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682518    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5e3e67-508a-4d81-851b-656fa138e61d" containerName="cleanup"
Nov 02 10:15:43.682755 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682523    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da7b91c-3ac1-46b9-a47f-347e8775cf4b" containerName="payload-processing"
Nov 02 10:15:43.682755 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682530    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5e3e67-508a-4d81-851b-656fa138e61d" containerName="cleanup"
Nov 02 10:15:43.682755 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.682536    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00e3a95-1a6e-411e-b928-456a968e432e" containerName="istio-proxy"
Nov 02 10:15:43.686355 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.686338    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-75fb58b5b7-xbsfb"
Nov 02 10:15:43.691746 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.691725    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 10:15:43.693176 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.693144    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ftlll\""
Nov 02 10:15:43.695878 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.695857    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-75fb58b5b7-xbsfb"]
Nov 02 10:15:43.755805 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.755767    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2smj6\" (UniqueName: \"kubernetes.io/projected/3b4a41e8-9183-4a0a-b088-4a292363401f-kube-api-access-2smj6\") pod \"payload-processing-75fb58b5b7-xbsfb\" (UID: \"3b4a41e8-9183-4a0a-b088-4a292363401f\") " pod="openshift-ingress/payload-processing-75fb58b5b7-xbsfb"
Nov 02 10:15:43.755805 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.755805    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4a41e8-9183-4a0a-b088-4a292363401f-plugins-config-volume\") pod \"payload-processing-75fb58b5b7-xbsfb\" (UID: \"3b4a41e8-9183-4a0a-b088-4a292363401f\") " pod="openshift-ingress/payload-processing-75fb58b5b7-xbsfb"
Nov 02 10:15:43.856640 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.856606    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2smj6\" (UniqueName: \"kubernetes.io/projected/3b4a41e8-9183-4a0a-b088-4a292363401f-kube-api-access-2smj6\") pod \"payload-processing-75fb58b5b7-xbsfb\" (UID: \"3b4a41e8-9183-4a0a-b088-4a292363401f\") " pod="openshift-ingress/payload-processing-75fb58b5b7-xbsfb"
Nov 02 10:15:43.856640 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.856643    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4a41e8-9183-4a0a-b088-4a292363401f-plugins-config-volume\") pod \"payload-processing-75fb58b5b7-xbsfb\" (UID: \"3b4a41e8-9183-4a0a-b088-4a292363401f\") " pod="openshift-ingress/payload-processing-75fb58b5b7-xbsfb"
Nov 02 10:15:43.857392 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.857371    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4a41e8-9183-4a0a-b088-4a292363401f-plugins-config-volume\") pod \"payload-processing-75fb58b5b7-xbsfb\" (UID: \"3b4a41e8-9183-4a0a-b088-4a292363401f\") " pod="openshift-ingress/payload-processing-75fb58b5b7-xbsfb"
Nov 02 10:15:43.874586 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:43.874551    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2smj6\" (UniqueName: \"kubernetes.io/projected/3b4a41e8-9183-4a0a-b088-4a292363401f-kube-api-access-2smj6\") pod \"payload-processing-75fb58b5b7-xbsfb\" (UID: \"3b4a41e8-9183-4a0a-b088-4a292363401f\") " pod="openshift-ingress/payload-processing-75fb58b5b7-xbsfb"
Nov 02 10:15:44.002114 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:44.002030    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-75fb58b5b7-xbsfb"
Nov 02 10:15:44.278932 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:44.278905    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-75fb58b5b7-xbsfb"]
Nov 02 10:15:44.281031 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:15:44.281005    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4a41e8_9183_4a0a_b088_4a292363401f.slice/crio-2409e8344421bdc919732d1cfd6861883e0cc9afeb1c452507bb1bff92092dd9 WatchSource:0}: Error finding container 2409e8344421bdc919732d1cfd6861883e0cc9afeb1c452507bb1bff92092dd9: Status 404 returned error can't find the container with id 2409e8344421bdc919732d1cfd6861883e0cc9afeb1c452507bb1bff92092dd9
Nov 02 10:15:44.721293 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:44.721254    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-75fb58b5b7-xbsfb" event={"ID":"3b4a41e8-9183-4a0a-b088-4a292363401f","Type":"ContainerStarted","Data":"884218326f848c572a47e0717849f5f490c666c1671521820630481d03958660"}
Nov 02 10:15:44.721713 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:44.721319    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-75fb58b5b7-xbsfb" event={"ID":"3b4a41e8-9183-4a0a-b088-4a292363401f","Type":"ContainerStarted","Data":"2409e8344421bdc919732d1cfd6861883e0cc9afeb1c452507bb1bff92092dd9"}
Nov 02 10:15:44.721713 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:44.721386    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-75fb58b5b7-xbsfb"
Nov 02 10:15:44.739956 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:44.739900    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-75fb58b5b7-xbsfb" podStartSLOduration=1.739886745 podStartE2EDuration="1.739886745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:15:44.739266018 +0000 UTC m=+2421.677552794" watchObservedRunningTime="2026-11-02 10:15:44.739886745 +0000 UTC m=+2421.678173520"
Nov 02 10:15:55.729973 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:15:55.729935    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-75fb58b5b7-xbsfb"
Nov 02 10:16:03.643431 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:16:03.643377    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-24rkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:16:03.643973 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:16:03.643447    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:16:06.547074 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:16:06.547040    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:16:26.027031 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:16:26.026998    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"]
Nov 02 10:16:26.030756 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:16:26.030727    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4hqp9"]
Nov 02 10:16:26.473427 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:16:26.473390    4449 scope.go:122] "RemoveContainer" containerID="45458c6b9789df63c55a7d8dbe62b9ea5ca3c83593946a6931ae48d3d580d056"
Nov 02 10:16:27.205124 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:16:27.205090    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12dea4ac-9dd2-483b-b57b-285ef0a56592" path="/var/lib/kubelet/pods/12dea4ac-9dd2-483b-b57b-285ef0a56592/volumes"
Nov 02 10:16:36.547077 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:16:36.546985    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:16:44.028365 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:16:44.028328    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"]
Nov 02 10:16:44.031182 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:16:44.031151    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4rfzs"]
Nov 02 10:16:45.205779 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:16:45.205740    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29484203-c406-4f33-b328-0162e1472972" path="/var/lib/kubelet/pods/29484203-c406-4f33-b328-0162e1472972/volumes"
Nov 02 10:17:02.025318 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:02.025277    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"]
Nov 02 10:17:02.031040 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:02.031010    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2195d4e8-2l6ns"]
Nov 02 10:17:03.207508 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:03.207471    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f" path="/var/lib/kubelet/pods/a25e45fb-a71f-4492-9dfa-5eb60e5a9f0f/volumes"
Nov 02 10:17:06.546933 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:17:06.546896    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:17:20.911767 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:20.911727    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"]
Nov 02 10:17:20.912245 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:20.912209    4449 cpu_manager.go:414] "containerMap: removing container" podUID="c00e3a95-1a6e-411e-b928-456a968e432e" containerName="istio-proxy"
Nov 02 10:17:20.912303 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:20.912246    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00e3a95-1a6e-411e-b928-456a968e432e" containerName="istio-proxy"
Nov 02 10:17:20.915479 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:20.915463    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"
Nov 02 10:17:20.921746 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:20.921719    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:17:20.923066 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:20.923037    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:17:20.923193 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:20.923079    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:17:20.923193 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:20.923101    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:17:20.926809 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:20.926786    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"]
Nov 02 10:17:20.987869 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:20.987823    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9v46\" (UniqueName: \"kubernetes.io/projected/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-kube-api-access-m9v46\") pod \"maas-api-revoke-sub-e2547622-nf7kf\" (UID: \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"
Nov 02 10:17:20.988061 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:20.987951    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-nf7kf\" (UID: \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"
Nov 02 10:17:20.988114 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:20.988060    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-nf7kf\" (UID: \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"
Nov 02 10:17:21.088784 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:21.088749    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-nf7kf\" (UID: \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"
Nov 02 10:17:21.088960 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:21.088796    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9v46\" (UniqueName: \"kubernetes.io/projected/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-kube-api-access-m9v46\") pod \"maas-api-revoke-sub-e2547622-nf7kf\" (UID: \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"
Nov 02 10:17:21.088960 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:21.088864    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-nf7kf\" (UID: \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"
Nov 02 10:17:21.089670 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:21.089647    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-nf7kf\" (UID: \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"
Nov 02 10:17:21.091563 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:21.091535    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-nf7kf\" (UID: \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"
Nov 02 10:17:21.101834 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:21.101809    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9v46\" (UniqueName: \"kubernetes.io/projected/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-kube-api-access-m9v46\") pod \"maas-api-revoke-sub-e2547622-nf7kf\" (UID: \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"
Nov 02 10:17:21.229511 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:21.229432    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"
Nov 02 10:17:21.508653 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:21.508626    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"]
Nov 02 10:17:21.510322 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:17:21.510295    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c45a9c5_2592_4fb7_9807_bfa772dd83fa.slice/crio-0de86764b6cf4437a16ebc0a30dd3f10c64662232cd934a64f085025515f83f0 WatchSource:0}: Error finding container 0de86764b6cf4437a16ebc0a30dd3f10c64662232cd934a64f085025515f83f0: Status 404 returned error can't find the container with id 0de86764b6cf4437a16ebc0a30dd3f10c64662232cd934a64f085025515f83f0
Nov 02 10:17:22.299231 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:22.299191    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf" event={"ID":"4c45a9c5-2592-4fb7-9807-bfa772dd83fa","Type":"ContainerStarted","Data":"b1bcb40e2530f80719c8dc65e85d9843c168be03514e4b948102effc12b1ccf8"}
Nov 02 10:17:22.299641 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:22.299257    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf" event={"ID":"4c45a9c5-2592-4fb7-9807-bfa772dd83fa","Type":"ContainerStarted","Data":"0de86764b6cf4437a16ebc0a30dd3f10c64662232cd934a64f085025515f83f0"}
Nov 02 10:17:22.317639 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:22.317591    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf" podStartSLOduration=2.317578054 podStartE2EDuration="2.317578054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:17:22.317397233 +0000 UTC m=+2519.255684025" watchObservedRunningTime="2026-11-02 10:17:22.317578054 +0000 UTC m=+2519.255864830"
Nov 02 10:17:23.306508 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:23.306471    4449 generic.go:356] "Generic (PLEG): container finished" podID="4c45a9c5-2592-4fb7-9807-bfa772dd83fa" containerID="b1bcb40e2530f80719c8dc65e85d9843c168be03514e4b948102effc12b1ccf8" exitCode=0
Nov 02 10:17:23.306914 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:23.306548    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf" event={"ID":"4c45a9c5-2592-4fb7-9807-bfa772dd83fa","Type":"ContainerDied","Data":"b1bcb40e2530f80719c8dc65e85d9843c168be03514e4b948102effc12b1ccf8"}
Nov 02 10:17:24.455622 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:24.455593    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"
Nov 02 10:17:24.524297 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:24.524265    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-openshift-ca\") pod \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\" (UID: \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\") "
Nov 02 10:17:24.524500 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:24.524359    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-xks-ca\") pod \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\" (UID: \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\") "
Nov 02 10:17:24.524500 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:24.524416    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-kube-api-access-m9v46\" (UniqueName: \"kubernetes.io/projected/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-kube-api-access-m9v46\") pod \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\" (UID: \"4c45a9c5-2592-4fb7-9807-bfa772dd83fa\") "
Nov 02 10:17:24.524647 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:24.524623    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-openshift-ca" pod "4c45a9c5-2592-4fb7-9807-bfa772dd83fa" (UID: "4c45a9c5-2592-4fb7-9807-bfa772dd83fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:17:24.524738 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:24.524725    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:17:24.526616 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:24.526590    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-kube-api-access-m9v46" pod "4c45a9c5-2592-4fb7-9807-bfa772dd83fa" (UID: "4c45a9c5-2592-4fb7-9807-bfa772dd83fa"). InnerVolumeSpecName "kube-api-access-m9v46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:17:24.526775 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:24.526754    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-xks-ca" pod "4c45a9c5-2592-4fb7-9807-bfa772dd83fa" (UID: "4c45a9c5-2592-4fb7-9807-bfa772dd83fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:17:24.625802 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:24.625693    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:17:24.625802 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:24.625744    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9v46\" (UniqueName: \"kubernetes.io/projected/4c45a9c5-2592-4fb7-9807-bfa772dd83fa-kube-api-access-m9v46\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:17:25.320372 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:25.320334    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf" event={"ID":"4c45a9c5-2592-4fb7-9807-bfa772dd83fa","Type":"ContainerDied","Data":"0de86764b6cf4437a16ebc0a30dd3f10c64662232cd934a64f085025515f83f0"}
Nov 02 10:17:25.320557 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:25.320391    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0de86764b6cf4437a16ebc0a30dd3f10c64662232cd934a64f085025515f83f0"
Nov 02 10:17:25.320557 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:25.320357    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"
Nov 02 10:17:26.500912 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:26.500884    4449 scope.go:122] "RemoveContainer" containerID="beef61f35b2475a36a899362afb7c835c733f1c842357a12e52301388f15fe31"
Nov 02 10:17:36.547045 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:17:36.547003    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:17:37.851914 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.851882    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"]
Nov 02 10:17:37.852664 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.852639    4449 cpu_manager.go:414] "containerMap: removing container" podUID="4c45a9c5-2592-4fb7-9807-bfa772dd83fa" containerName="revoke-keys"
Nov 02 10:17:37.852664 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.852665    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c45a9c5-2592-4fb7-9807-bfa772dd83fa" containerName="revoke-keys"
Nov 02 10:17:37.852864 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.852791    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c45a9c5-2592-4fb7-9807-bfa772dd83fa" containerName="revoke-keys"
Nov 02 10:17:37.857356 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.857336    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"
Nov 02 10:17:37.870259 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.870207    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hnhvm\" (UID: \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"
Nov 02 10:17:37.870469 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.870445    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hnhvm\" (UID: \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"
Nov 02 10:17:37.870705 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.870683    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5xtd\" (UniqueName: \"kubernetes.io/projected/ca92f266-e97e-4009-9cc6-5d138a1c9a92-kube-api-access-q5xtd\") pod \"maas-api-revoke-sub-c1e271a9-hnhvm\" (UID: \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"
Nov 02 10:17:37.871275 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.871253    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:17:37.871553 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.871530    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:17:37.871667 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.871600    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:17:37.871667 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.871651    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:17:37.873819 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.873793    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"]
Nov 02 10:17:37.973802 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.971862    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hnhvm\" (UID: \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"
Nov 02 10:17:37.973802 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.971930    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hnhvm\" (UID: \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"
Nov 02 10:17:37.973802 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.971971    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5xtd\" (UniqueName: \"kubernetes.io/projected/ca92f266-e97e-4009-9cc6-5d138a1c9a92-kube-api-access-q5xtd\") pod \"maas-api-revoke-sub-c1e271a9-hnhvm\" (UID: \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"
Nov 02 10:17:37.973802 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:37.973387    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hnhvm\" (UID: \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"
Nov 02 10:17:38.005131 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:38.005059    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5xtd\" (UniqueName: \"kubernetes.io/projected/ca92f266-e97e-4009-9cc6-5d138a1c9a92-kube-api-access-q5xtd\") pod \"maas-api-revoke-sub-c1e271a9-hnhvm\" (UID: \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"
Nov 02 10:17:38.030124 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:38.029722    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hnhvm\" (UID: \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"
Nov 02 10:17:38.197294 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:38.196355    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"
Nov 02 10:17:38.621530 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:38.621495    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"]
Nov 02 10:17:39.540891 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:39.540849    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm" event={"ID":"ca92f266-e97e-4009-9cc6-5d138a1c9a92","Type":"ContainerStarted","Data":"2c4f522eb03baec0a397b2ba3a62d88b95b3ccc281c17d16fff670ee355de376"}
Nov 02 10:17:39.541354 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:39.540946    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm" event={"ID":"ca92f266-e97e-4009-9cc6-5d138a1c9a92","Type":"ContainerStarted","Data":"271898aeb0d15789a0733edb6080b40e3e2057872253fa6697a6e5cfc511c41d"}
Nov 02 10:17:39.579242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:39.565730    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm" podStartSLOduration=2.5657110899999997 podStartE2EDuration="2.56571109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:17:39.565469826 +0000 UTC m=+2536.503756632" watchObservedRunningTime="2026-11-02 10:17:39.56571109 +0000 UTC m=+2536.503997867"
Nov 02 10:17:40.548957 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:40.548908    4449 generic.go:356] "Generic (PLEG): container finished" podID="ca92f266-e97e-4009-9cc6-5d138a1c9a92" containerID="2c4f522eb03baec0a397b2ba3a62d88b95b3ccc281c17d16fff670ee355de376" exitCode=0
Nov 02 10:17:40.549416 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:40.548992    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm" event={"ID":"ca92f266-e97e-4009-9cc6-5d138a1c9a92","Type":"ContainerDied","Data":"2c4f522eb03baec0a397b2ba3a62d88b95b3ccc281c17d16fff670ee355de376"}
Nov 02 10:17:41.691310 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:41.691281    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"
Nov 02 10:17:41.759792 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:41.759757    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca92f266-e97e-4009-9cc6-5d138a1c9a92-kube-api-access-q5xtd\" (UniqueName: \"kubernetes.io/projected/ca92f266-e97e-4009-9cc6-5d138a1c9a92-kube-api-access-q5xtd\") pod \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\" (UID: \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\") "
Nov 02 10:17:41.760001 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:41.759824    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-xks-ca\") pod \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\" (UID: \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\") "
Nov 02 10:17:41.760001 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:41.759860    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-openshift-ca\") pod \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\" (UID: \"ca92f266-e97e-4009-9cc6-5d138a1c9a92\") "
Nov 02 10:17:41.760320 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:41.760290    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-openshift-ca" pod "ca92f266-e97e-4009-9cc6-5d138a1c9a92" (UID: "ca92f266-e97e-4009-9cc6-5d138a1c9a92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:17:41.762155 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:41.762129    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-xks-ca" pod "ca92f266-e97e-4009-9cc6-5d138a1c9a92" (UID: "ca92f266-e97e-4009-9cc6-5d138a1c9a92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:17:41.762295 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:41.762157    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca92f266-e97e-4009-9cc6-5d138a1c9a92-kube-api-access-q5xtd" pod "ca92f266-e97e-4009-9cc6-5d138a1c9a92" (UID: "ca92f266-e97e-4009-9cc6-5d138a1c9a92"). InnerVolumeSpecName "kube-api-access-q5xtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:17:41.861375 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:41.861283    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:17:41.861375 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:41.861318    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5xtd\" (UniqueName: \"kubernetes.io/projected/ca92f266-e97e-4009-9cc6-5d138a1c9a92-kube-api-access-q5xtd\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:17:41.861375 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:41.861333    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca92f266-e97e-4009-9cc6-5d138a1c9a92-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:17:42.565007 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:42.564977    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"
Nov 02 10:17:42.565194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:42.565012    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm" event={"ID":"ca92f266-e97e-4009-9cc6-5d138a1c9a92","Type":"ContainerDied","Data":"271898aeb0d15789a0733edb6080b40e3e2057872253fa6697a6e5cfc511c41d"}
Nov 02 10:17:42.565194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:42.565060    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="271898aeb0d15789a0733edb6080b40e3e2057872253fa6697a6e5cfc511c41d"
Nov 02 10:17:55.364574 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.364538    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"]
Nov 02 10:17:55.364984 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.364971    4449 cpu_manager.go:414] "containerMap: removing container" podUID="ca92f266-e97e-4009-9cc6-5d138a1c9a92" containerName="revoke-keys"
Nov 02 10:17:55.364984 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.364981    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca92f266-e97e-4009-9cc6-5d138a1c9a92" containerName="revoke-keys"
Nov 02 10:17:55.365078 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.365068    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca92f266-e97e-4009-9cc6-5d138a1c9a92" containerName="revoke-keys"
Nov 02 10:17:55.368544 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.368519    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"
Nov 02 10:17:55.379043 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.379016    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:17:55.379232 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.379039    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:17:55.382423 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.382400    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"]
Nov 02 10:17:55.403165 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.403130    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:17:55.403378 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.403229    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:17:55.438717 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.438685    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6hthp\" (UID: \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"
Nov 02 10:17:55.438876 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.438757    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6hthp\" (UID: \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"
Nov 02 10:17:55.438919 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.438877    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6hh5\" (UniqueName: \"kubernetes.io/projected/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-kube-api-access-l6hh5\") pod \"maas-api-revoke-sub-2e1f5b5f-6hthp\" (UID: \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"
Nov 02 10:17:55.540320 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.540279    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6hthp\" (UID: \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"
Nov 02 10:17:55.540513 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.540349    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6hthp\" (UID: \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"
Nov 02 10:17:55.540513 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.540421    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6hh5\" (UniqueName: \"kubernetes.io/projected/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-kube-api-access-l6hh5\") pod \"maas-api-revoke-sub-2e1f5b5f-6hthp\" (UID: \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"
Nov 02 10:17:55.541013 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.540988    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6hthp\" (UID: \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"
Nov 02 10:17:55.542925 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.542892    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6hthp\" (UID: \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"
Nov 02 10:17:55.553876 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.553851    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6hh5\" (UniqueName: \"kubernetes.io/projected/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-kube-api-access-l6hh5\") pod \"maas-api-revoke-sub-2e1f5b5f-6hthp\" (UID: \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"
Nov 02 10:17:55.685316 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.685205    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"
Nov 02 10:17:55.961957 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:55.961925    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"]
Nov 02 10:17:55.964120 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:17:55.964093    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b561a93_0f0a_40f7_adf4_b3bfcf60ccb2.slice/crio-5df584f58cca130fb61a5d0fc3c3b7e942a2fd9da31340420b4abfe970fc80b7 WatchSource:0}: Error finding container 5df584f58cca130fb61a5d0fc3c3b7e942a2fd9da31340420b4abfe970fc80b7: Status 404 returned error can't find the container with id 5df584f58cca130fb61a5d0fc3c3b7e942a2fd9da31340420b4abfe970fc80b7
Nov 02 10:17:56.658143 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:56.658104    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp" event={"ID":"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2","Type":"ContainerStarted","Data":"c8dc3d5a91e541d3d750ddb1874b3c2c29113caac713709e5c0f6ffec0d2a2ff"}
Nov 02 10:17:56.658603 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:56.658161    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp" event={"ID":"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2","Type":"ContainerStarted","Data":"5df584f58cca130fb61a5d0fc3c3b7e942a2fd9da31340420b4abfe970fc80b7"}
Nov 02 10:17:56.680262 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:56.680191    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp" podStartSLOduration=1.680177397 podStartE2EDuration="1.680177397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:17:56.679273462 +0000 UTC m=+2553.617560253" watchObservedRunningTime="2026-11-02 10:17:56.680177397 +0000 UTC m=+2553.618464172"
Nov 02 10:17:57.665870 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:57.665829    4449 generic.go:356] "Generic (PLEG): container finished" podID="9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2" containerID="c8dc3d5a91e541d3d750ddb1874b3c2c29113caac713709e5c0f6ffec0d2a2ff" exitCode=0
Nov 02 10:17:57.666454 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:57.665920    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp" event={"ID":"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2","Type":"ContainerDied","Data":"c8dc3d5a91e541d3d750ddb1874b3c2c29113caac713709e5c0f6ffec0d2a2ff"}
Nov 02 10:17:58.818068 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:58.818037    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"
Nov 02 10:17:58.974621 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:58.974525    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-openshift-ca\") pod \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\" (UID: \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\") "
Nov 02 10:17:58.974809 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:58.974622    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-kube-api-access-l6hh5\" (UniqueName: \"kubernetes.io/projected/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-kube-api-access-l6hh5\") pod \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\" (UID: \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\") "
Nov 02 10:17:58.974809 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:58.974690    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-xks-ca\") pod \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\" (UID: \"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2\") "
Nov 02 10:17:58.974951 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:58.974925    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-openshift-ca" pod "9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2" (UID: "9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:17:58.975090 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:58.975075    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:17:58.976934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:58.976910    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-xks-ca" pod "9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2" (UID: "9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:17:58.977006 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:58.976959    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-kube-api-access-l6hh5" pod "9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2" (UID: "9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2"). InnerVolumeSpecName "kube-api-access-l6hh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:17:59.075995 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:59.075959    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6hh5\" (UniqueName: \"kubernetes.io/projected/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-kube-api-access-l6hh5\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:17:59.075995 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:59.075993    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:17:59.677715 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:59.677604    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"
Nov 02 10:17:59.677715 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:59.677643    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp" event={"ID":"9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2","Type":"ContainerDied","Data":"5df584f58cca130fb61a5d0fc3c3b7e942a2fd9da31340420b4abfe970fc80b7"}
Nov 02 10:17:59.677715 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:17:59.677694    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5df584f58cca130fb61a5d0fc3c3b7e942a2fd9da31340420b4abfe970fc80b7"
Nov 02 10:18:06.371263 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.371169    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"]
Nov 02 10:18:06.371691 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.371673    4449 cpu_manager.go:414] "containerMap: removing container" podUID="9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2" containerName="revoke-keys"
Nov 02 10:18:06.371735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.371693    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2" containerName="revoke-keys"
Nov 02 10:18:06.371801 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.371791    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2" containerName="revoke-keys"
Nov 02 10:18:06.375178 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.375149    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"
Nov 02 10:18:06.382097 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.382065    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:18:06.383497 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.382732    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:18:06.383497 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.382810    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:18:06.383497 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.382856    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:18:06.384850 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.384822    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"]
Nov 02 10:18:06.547315 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:18:06.547277    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:18:06.554038 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.553998    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadfacd0-e678-42c4-8188-692802062b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-qbg69\" (UID: \"fadfacd0-e678-42c4-8188-692802062b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"
Nov 02 10:18:06.554264 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.554204    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smkr2\" (UniqueName: \"kubernetes.io/projected/fadfacd0-e678-42c4-8188-692802062b74-kube-api-access-smkr2\") pod \"maas-api-revoke-sub-ec843018-qbg69\" (UID: \"fadfacd0-e678-42c4-8188-692802062b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"
Nov 02 10:18:06.554339 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.554321    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadfacd0-e678-42c4-8188-692802062b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-qbg69\" (UID: \"fadfacd0-e678-42c4-8188-692802062b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"
Nov 02 10:18:06.655529 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.655445    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smkr2\" (UniqueName: \"kubernetes.io/projected/fadfacd0-e678-42c4-8188-692802062b74-kube-api-access-smkr2\") pod \"maas-api-revoke-sub-ec843018-qbg69\" (UID: \"fadfacd0-e678-42c4-8188-692802062b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"
Nov 02 10:18:06.655529 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.655495    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadfacd0-e678-42c4-8188-692802062b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-qbg69\" (UID: \"fadfacd0-e678-42c4-8188-692802062b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"
Nov 02 10:18:06.655787 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.655545    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadfacd0-e678-42c4-8188-692802062b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-qbg69\" (UID: \"fadfacd0-e678-42c4-8188-692802062b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"
Nov 02 10:18:06.656191 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.656155    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadfacd0-e678-42c4-8188-692802062b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-qbg69\" (UID: \"fadfacd0-e678-42c4-8188-692802062b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"
Nov 02 10:18:06.657982 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.657963    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadfacd0-e678-42c4-8188-692802062b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-qbg69\" (UID: \"fadfacd0-e678-42c4-8188-692802062b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"
Nov 02 10:18:06.668303 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.668266    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smkr2\" (UniqueName: \"kubernetes.io/projected/fadfacd0-e678-42c4-8188-692802062b74-kube-api-access-smkr2\") pod \"maas-api-revoke-sub-ec843018-qbg69\" (UID: \"fadfacd0-e678-42c4-8188-692802062b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"
Nov 02 10:18:06.691050 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.691008    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"
Nov 02 10:18:06.988567 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:06.988531    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"]
Nov 02 10:18:06.990441 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:18:06.990415    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfadfacd0_e678_42c4_8188_692802062b74.slice/crio-b1b35a513e6ee538c806e95eac0d446cdae6491788ff76ab80721a207c2ab399 WatchSource:0}: Error finding container b1b35a513e6ee538c806e95eac0d446cdae6491788ff76ab80721a207c2ab399: Status 404 returned error can't find the container with id b1b35a513e6ee538c806e95eac0d446cdae6491788ff76ab80721a207c2ab399
Nov 02 10:18:07.743356 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:07.743307    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69" event={"ID":"fadfacd0-e678-42c4-8188-692802062b74","Type":"ContainerStarted","Data":"501a9fa9123c80921d7ab61d66932853cc42a50e7555f63a97e2233064d4aad2"}
Nov 02 10:18:07.743783 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:07.743376    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69" event={"ID":"fadfacd0-e678-42c4-8188-692802062b74","Type":"ContainerStarted","Data":"b1b35a513e6ee538c806e95eac0d446cdae6491788ff76ab80721a207c2ab399"}
Nov 02 10:18:07.764045 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:07.763986    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69" podStartSLOduration=1.763965375 podStartE2EDuration="1.763965375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:18:07.76186658 +0000 UTC m=+2564.700153356" watchObservedRunningTime="2026-11-02 10:18:07.763965375 +0000 UTC m=+2564.702252152"
Nov 02 10:18:08.754567 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:08.754529    4449 generic.go:356] "Generic (PLEG): container finished" podID="fadfacd0-e678-42c4-8188-692802062b74" containerID="501a9fa9123c80921d7ab61d66932853cc42a50e7555f63a97e2233064d4aad2" exitCode=0
Nov 02 10:18:08.755116 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:08.754619    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69" event={"ID":"fadfacd0-e678-42c4-8188-692802062b74","Type":"ContainerDied","Data":"501a9fa9123c80921d7ab61d66932853cc42a50e7555f63a97e2233064d4aad2"}
Nov 02 10:18:10.229425 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:10.229400    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"
Nov 02 10:18:10.326616 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:10.326573    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fadfacd0-e678-42c4-8188-692802062b74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadfacd0-e678-42c4-8188-692802062b74-maas-api-xks-ca\") pod \"fadfacd0-e678-42c4-8188-692802062b74\" (UID: \"fadfacd0-e678-42c4-8188-692802062b74\") "
Nov 02 10:18:10.331489 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:10.331445    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fadfacd0-e678-42c4-8188-692802062b74-kube-api-access-smkr2\" (UniqueName: \"kubernetes.io/projected/fadfacd0-e678-42c4-8188-692802062b74-kube-api-access-smkr2\") pod \"fadfacd0-e678-42c4-8188-692802062b74\" (UID: \"fadfacd0-e678-42c4-8188-692802062b74\") "
Nov 02 10:18:10.331649 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:10.331542    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fadfacd0-e678-42c4-8188-692802062b74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadfacd0-e678-42c4-8188-692802062b74-maas-api-openshift-ca\") pod \"fadfacd0-e678-42c4-8188-692802062b74\" (UID: \"fadfacd0-e678-42c4-8188-692802062b74\") "
Nov 02 10:18:10.332459 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:10.332416    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadfacd0-e678-42c4-8188-692802062b74-maas-api-openshift-ca" pod "fadfacd0-e678-42c4-8188-692802062b74" (UID: "fadfacd0-e678-42c4-8188-692802062b74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:18:10.333999 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:10.333960    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fadfacd0-e678-42c4-8188-692802062b74-maas-api-xks-ca" pod "fadfacd0-e678-42c4-8188-692802062b74" (UID: "fadfacd0-e678-42c4-8188-692802062b74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:18:10.334978 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:10.334955    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadfacd0-e678-42c4-8188-692802062b74-kube-api-access-smkr2" pod "fadfacd0-e678-42c4-8188-692802062b74" (UID: "fadfacd0-e678-42c4-8188-692802062b74"). InnerVolumeSpecName "kube-api-access-smkr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:18:10.432810 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:10.432710    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadfacd0-e678-42c4-8188-692802062b74-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:18:10.432810 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:10.432745    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadfacd0-e678-42c4-8188-692802062b74-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:18:10.432810 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:10.432760    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smkr2\" (UniqueName: \"kubernetes.io/projected/fadfacd0-e678-42c4-8188-692802062b74-kube-api-access-smkr2\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:18:10.828120 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:10.828079    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69" event={"ID":"fadfacd0-e678-42c4-8188-692802062b74","Type":"ContainerDied","Data":"b1b35a513e6ee538c806e95eac0d446cdae6491788ff76ab80721a207c2ab399"}
Nov 02 10:18:10.828501 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:10.828479    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1b35a513e6ee538c806e95eac0d446cdae6491788ff76ab80721a207c2ab399"
Nov 02 10:18:10.828704 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:10.828688    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"
Nov 02 10:18:21.816376 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:21.816330    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:18:21.817036 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:21.816995    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:18:26.523275 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:26.523241    4449 scope.go:122] "RemoveContainer" containerID="052b99e01f0e850448eda14a3d54e1ebd123d845494e0f619e7eb277117713cd"
Nov 02 10:18:26.532705 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:26.532680    4449 scope.go:122] "RemoveContainer" containerID="663612cf145dce4fe544360f6867b4380752a4442cf9aa48126e2e1af02c8741"
Nov 02 10:18:32.990592 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:32.990552    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"]
Nov 02 10:18:32.991301 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:32.991278    4449 cpu_manager.go:414] "containerMap: removing container" podUID="fadfacd0-e678-42c4-8188-692802062b74" containerName="revoke-keys"
Nov 02 10:18:32.991568 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:32.991304    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fadfacd0-e678-42c4-8188-692802062b74" containerName="revoke-keys"
Nov 02 10:18:32.991568 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:32.991455    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="fadfacd0-e678-42c4-8188-692802062b74" containerName="revoke-keys"
Nov 02 10:18:32.996399 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:32.996376    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"
Nov 02 10:18:33.002899 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.002876    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:18:33.003029 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.002903    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:18:33.003029 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.002920    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:18:33.003029 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.002988    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:18:33.007173 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.007146    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"]
Nov 02 10:18:33.120073 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.120025    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5rsw\" (UniqueName: \"kubernetes.io/projected/5882afca-a46d-41d7-9c03-766f58607c01-kube-api-access-q5rsw\") pod \"maas-api-revoke-sub-046e1343-gn452\" (UID: \"5882afca-a46d-41d7-9c03-766f58607c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"
Nov 02 10:18:33.120073 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.120078    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-gn452\" (UID: \"5882afca-a46d-41d7-9c03-766f58607c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"
Nov 02 10:18:33.120330 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.120154    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-gn452\" (UID: \"5882afca-a46d-41d7-9c03-766f58607c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"
Nov 02 10:18:33.221470 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.221432    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5rsw\" (UniqueName: \"kubernetes.io/projected/5882afca-a46d-41d7-9c03-766f58607c01-kube-api-access-q5rsw\") pod \"maas-api-revoke-sub-046e1343-gn452\" (UID: \"5882afca-a46d-41d7-9c03-766f58607c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"
Nov 02 10:18:33.221470 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.221473    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-gn452\" (UID: \"5882afca-a46d-41d7-9c03-766f58607c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"
Nov 02 10:18:33.221761 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.221492    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-gn452\" (UID: \"5882afca-a46d-41d7-9c03-766f58607c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"
Nov 02 10:18:33.222137 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.222113    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-gn452\" (UID: \"5882afca-a46d-41d7-9c03-766f58607c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"
Nov 02 10:18:33.224036 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.224014    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-gn452\" (UID: \"5882afca-a46d-41d7-9c03-766f58607c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"
Nov 02 10:18:33.234873 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.234849    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5rsw\" (UniqueName: \"kubernetes.io/projected/5882afca-a46d-41d7-9c03-766f58607c01-kube-api-access-q5rsw\") pod \"maas-api-revoke-sub-046e1343-gn452\" (UID: \"5882afca-a46d-41d7-9c03-766f58607c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"
Nov 02 10:18:33.311462 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.311423    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"
Nov 02 10:18:33.595422 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:33.595388    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"]
Nov 02 10:18:33.596708 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:18:33.596682    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5882afca_a46d_41d7_9c03_766f58607c01.slice/crio-71cfba5763789a8e0513ac41ab79aa6ee490137e280c373863363a97b1ebaf9a WatchSource:0}: Error finding container 71cfba5763789a8e0513ac41ab79aa6ee490137e280c373863363a97b1ebaf9a: Status 404 returned error can't find the container with id 71cfba5763789a8e0513ac41ab79aa6ee490137e280c373863363a97b1ebaf9a
Nov 02 10:18:34.010849 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:34.010806    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452" event={"ID":"5882afca-a46d-41d7-9c03-766f58607c01","Type":"ContainerStarted","Data":"5456388dd3212cb42d3366f667f66bfee1fd90619a2bf279e7af885eee046d6b"}
Nov 02 10:18:34.011363 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:34.010863    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452" event={"ID":"5882afca-a46d-41d7-9c03-766f58607c01","Type":"ContainerStarted","Data":"71cfba5763789a8e0513ac41ab79aa6ee490137e280c373863363a97b1ebaf9a"}
Nov 02 10:18:34.033565 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:34.033510    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452" podStartSLOduration=2.033494205 podStartE2EDuration="2.033494205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:18:34.030833664 +0000 UTC m=+2590.969120464" watchObservedRunningTime="2026-11-02 10:18:34.033494205 +0000 UTC m=+2590.971780980"
Nov 02 10:18:35.017466 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:35.017428    4449 generic.go:356] "Generic (PLEG): container finished" podID="5882afca-a46d-41d7-9c03-766f58607c01" containerID="5456388dd3212cb42d3366f667f66bfee1fd90619a2bf279e7af885eee046d6b" exitCode=0
Nov 02 10:18:35.017890 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:35.017513    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452" event={"ID":"5882afca-a46d-41d7-9c03-766f58607c01","Type":"ContainerDied","Data":"5456388dd3212cb42d3366f667f66bfee1fd90619a2bf279e7af885eee046d6b"}
Nov 02 10:18:36.158719 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:36.158690    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"
Nov 02 10:18:36.251422 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:36.251389    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5882afca-a46d-41d7-9c03-766f58607c01-kube-api-access-q5rsw\" (UniqueName: \"kubernetes.io/projected/5882afca-a46d-41d7-9c03-766f58607c01-kube-api-access-q5rsw\") pod \"5882afca-a46d-41d7-9c03-766f58607c01\" (UID: \"5882afca-a46d-41d7-9c03-766f58607c01\") "
Nov 02 10:18:36.251632 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:36.251443    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-xks-ca\") pod \"5882afca-a46d-41d7-9c03-766f58607c01\" (UID: \"5882afca-a46d-41d7-9c03-766f58607c01\") "
Nov 02 10:18:36.251632 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:36.251555    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-openshift-ca\") pod \"5882afca-a46d-41d7-9c03-766f58607c01\" (UID: \"5882afca-a46d-41d7-9c03-766f58607c01\") "
Nov 02 10:18:36.251907 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:36.251882    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-openshift-ca" pod "5882afca-a46d-41d7-9c03-766f58607c01" (UID: "5882afca-a46d-41d7-9c03-766f58607c01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:18:36.253842 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:36.253810    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5882afca-a46d-41d7-9c03-766f58607c01-kube-api-access-q5rsw" pod "5882afca-a46d-41d7-9c03-766f58607c01" (UID: "5882afca-a46d-41d7-9c03-766f58607c01"). InnerVolumeSpecName "kube-api-access-q5rsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:18:36.253953 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:36.253865    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-xks-ca" pod "5882afca-a46d-41d7-9c03-766f58607c01" (UID: "5882afca-a46d-41d7-9c03-766f58607c01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:18:36.352665 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:36.352632    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5rsw\" (UniqueName: \"kubernetes.io/projected/5882afca-a46d-41d7-9c03-766f58607c01-kube-api-access-q5rsw\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:18:36.352665 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:36.352662    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:18:36.352665 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:36.352672    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5882afca-a46d-41d7-9c03-766f58607c01-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:18:36.545944 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:18:36.545907    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:18:37.030279 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:37.030248    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"
Nov 02 10:18:37.030462 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:37.030253    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452" event={"ID":"5882afca-a46d-41d7-9c03-766f58607c01","Type":"ContainerDied","Data":"71cfba5763789a8e0513ac41ab79aa6ee490137e280c373863363a97b1ebaf9a"}
Nov 02 10:18:37.030462 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:37.030380    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71cfba5763789a8e0513ac41ab79aa6ee490137e280c373863363a97b1ebaf9a"
Nov 02 10:18:43.232609 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.232575    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"]
Nov 02 10:18:43.233129 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.233113    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5882afca-a46d-41d7-9c03-766f58607c01" containerName="revoke-keys"
Nov 02 10:18:43.233179 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.233134    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5882afca-a46d-41d7-9c03-766f58607c01" containerName="revoke-keys"
Nov 02 10:18:43.233320 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.233243    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5882afca-a46d-41d7-9c03-766f58607c01" containerName="revoke-keys"
Nov 02 10:18:43.236778 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.236755    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"
Nov 02 10:18:43.247967 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.247933    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:18:43.249363 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.249338    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:18:43.249704 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.249683    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:18:43.250285 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.249773    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:18:43.252852 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.252827    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"]
Nov 02 10:18:43.322393 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.322357    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71952279-da18-405f-83b7-215b7cfc4540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rkrdt\" (UID: \"71952279-da18-405f-83b7-215b7cfc4540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"
Nov 02 10:18:43.322393 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.322400    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5ct\" (UniqueName: \"kubernetes.io/projected/71952279-da18-405f-83b7-215b7cfc4540-kube-api-access-8m5ct\") pod \"maas-api-revoke-sub-d3fba056-rkrdt\" (UID: \"71952279-da18-405f-83b7-215b7cfc4540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"
Nov 02 10:18:43.322643 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.322505    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71952279-da18-405f-83b7-215b7cfc4540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rkrdt\" (UID: \"71952279-da18-405f-83b7-215b7cfc4540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"
Nov 02 10:18:43.423446 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.423410    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71952279-da18-405f-83b7-215b7cfc4540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rkrdt\" (UID: \"71952279-da18-405f-83b7-215b7cfc4540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"
Nov 02 10:18:43.423446 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.423448    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5ct\" (UniqueName: \"kubernetes.io/projected/71952279-da18-405f-83b7-215b7cfc4540-kube-api-access-8m5ct\") pod \"maas-api-revoke-sub-d3fba056-rkrdt\" (UID: \"71952279-da18-405f-83b7-215b7cfc4540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"
Nov 02 10:18:43.423734 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.423482    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71952279-da18-405f-83b7-215b7cfc4540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rkrdt\" (UID: \"71952279-da18-405f-83b7-215b7cfc4540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"
Nov 02 10:18:43.424250 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.424167    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71952279-da18-405f-83b7-215b7cfc4540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rkrdt\" (UID: \"71952279-da18-405f-83b7-215b7cfc4540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"
Nov 02 10:18:43.426043 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.426020    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71952279-da18-405f-83b7-215b7cfc4540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rkrdt\" (UID: \"71952279-da18-405f-83b7-215b7cfc4540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"
Nov 02 10:18:43.436696 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.436675    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5ct\" (UniqueName: \"kubernetes.io/projected/71952279-da18-405f-83b7-215b7cfc4540-kube-api-access-8m5ct\") pod \"maas-api-revoke-sub-d3fba056-rkrdt\" (UID: \"71952279-da18-405f-83b7-215b7cfc4540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"
Nov 02 10:18:43.554391 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.554358    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"
Nov 02 10:18:43.852895 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:43.852870    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"]
Nov 02 10:18:44.069283 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:44.069242    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt" event={"ID":"71952279-da18-405f-83b7-215b7cfc4540","Type":"ContainerStarted","Data":"67d1f383d4628571ef874a2659331e6c855f15cba10e8151bfda406a7f0fc7e9"}
Nov 02 10:18:45.078399 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:45.078361    4449 generic.go:356] "Generic (PLEG): container finished" podID="71952279-da18-405f-83b7-215b7cfc4540" containerID="3dbb82abd54a60e615189971e338b5368e3e6bbf5e49302f284733f16dc41bb8" exitCode=0
Nov 02 10:18:45.078954 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:45.078419    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt" event={"ID":"71952279-da18-405f-83b7-215b7cfc4540","Type":"ContainerDied","Data":"3dbb82abd54a60e615189971e338b5368e3e6bbf5e49302f284733f16dc41bb8"}
Nov 02 10:18:46.481670 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:46.481636    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"
Nov 02 10:18:46.587244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:46.586616    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71952279-da18-405f-83b7-215b7cfc4540-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71952279-da18-405f-83b7-215b7cfc4540-maas-api-xks-ca\") pod \"71952279-da18-405f-83b7-215b7cfc4540\" (UID: \"71952279-da18-405f-83b7-215b7cfc4540\") "
Nov 02 10:18:46.587244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:46.586705    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71952279-da18-405f-83b7-215b7cfc4540-kube-api-access-8m5ct\" (UniqueName: \"kubernetes.io/projected/71952279-da18-405f-83b7-215b7cfc4540-kube-api-access-8m5ct\") pod \"71952279-da18-405f-83b7-215b7cfc4540\" (UID: \"71952279-da18-405f-83b7-215b7cfc4540\") "
Nov 02 10:18:46.587244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:46.586769    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71952279-da18-405f-83b7-215b7cfc4540-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71952279-da18-405f-83b7-215b7cfc4540-maas-api-openshift-ca\") pod \"71952279-da18-405f-83b7-215b7cfc4540\" (UID: \"71952279-da18-405f-83b7-215b7cfc4540\") "
Nov 02 10:18:46.625331 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:46.624484    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71952279-da18-405f-83b7-215b7cfc4540-maas-api-openshift-ca" pod "71952279-da18-405f-83b7-215b7cfc4540" (UID: "71952279-da18-405f-83b7-215b7cfc4540"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:18:46.625331 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:46.625074    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71952279-da18-405f-83b7-215b7cfc4540-maas-api-xks-ca" pod "71952279-da18-405f-83b7-215b7cfc4540" (UID: "71952279-da18-405f-83b7-215b7cfc4540"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:18:46.625331 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:46.625191    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71952279-da18-405f-83b7-215b7cfc4540-kube-api-access-8m5ct" pod "71952279-da18-405f-83b7-215b7cfc4540" (UID: "71952279-da18-405f-83b7-215b7cfc4540"). InnerVolumeSpecName "kube-api-access-8m5ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:18:46.690145 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:46.689946    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71952279-da18-405f-83b7-215b7cfc4540-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:18:46.690145 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:46.690009    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m5ct\" (UniqueName: \"kubernetes.io/projected/71952279-da18-405f-83b7-215b7cfc4540-kube-api-access-8m5ct\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:18:46.690145 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:46.690029    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71952279-da18-405f-83b7-215b7cfc4540-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:18:47.125394 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:47.125354    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"
Nov 02 10:18:47.125716 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:47.125395    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt" event={"ID":"71952279-da18-405f-83b7-215b7cfc4540","Type":"ContainerDied","Data":"67d1f383d4628571ef874a2659331e6c855f15cba10e8151bfda406a7f0fc7e9"}
Nov 02 10:18:47.125889 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:18:47.125873    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67d1f383d4628571ef874a2659331e6c855f15cba10e8151bfda406a7f0fc7e9"
Nov 02 10:19:04.075390 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.075351    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"]
Nov 02 10:19:04.075908 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.075792    4449 cpu_manager.go:414] "containerMap: removing container" podUID="71952279-da18-405f-83b7-215b7cfc4540" containerName="revoke-keys"
Nov 02 10:19:04.075908 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.075803    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71952279-da18-405f-83b7-215b7cfc4540" containerName="revoke-keys"
Nov 02 10:19:04.075908 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.075886    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="71952279-da18-405f-83b7-215b7cfc4540" containerName="revoke-keys"
Nov 02 10:19:04.079638 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.079615    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"
Nov 02 10:19:04.083622 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.083583    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:19:04.083778 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.083669    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:19:04.084886 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.084864    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:19:04.084886 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.084877    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:19:04.091095 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.091035    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"]
Nov 02 10:19:04.213203 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.213169    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-265k6\" (UID: \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"
Nov 02 10:19:04.213414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.213318    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnnv\" (UniqueName: \"kubernetes.io/projected/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-kube-api-access-flnnv\") pod \"maas-api-revoke-sub-ae594979-265k6\" (UID: \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"
Nov 02 10:19:04.213414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.213373    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-265k6\" (UID: \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"
Nov 02 10:19:04.314333 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.314296    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnnv\" (UniqueName: \"kubernetes.io/projected/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-kube-api-access-flnnv\") pod \"maas-api-revoke-sub-ae594979-265k6\" (UID: \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"
Nov 02 10:19:04.314520 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.314362    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-265k6\" (UID: \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"
Nov 02 10:19:04.314520 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.314417    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-265k6\" (UID: \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"
Nov 02 10:19:04.315196 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.315106    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-265k6\" (UID: \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"
Nov 02 10:19:04.316942 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.316918    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-265k6\" (UID: \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"
Nov 02 10:19:04.327613 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.327554    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnnv\" (UniqueName: \"kubernetes.io/projected/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-kube-api-access-flnnv\") pod \"maas-api-revoke-sub-ae594979-265k6\" (UID: \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"
Nov 02 10:19:04.397586 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.397548    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"
Nov 02 10:19:04.678667 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:04.678640    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"]
Nov 02 10:19:04.679684 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:19:04.679661    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a07b3dc_ccdf_41cc_88ca_c9d5475cf149.slice/crio-ce021deef11db514810c343a7c4f5b0d8082f60c8597c8b87ad684d822753f7b WatchSource:0}: Error finding container ce021deef11db514810c343a7c4f5b0d8082f60c8597c8b87ad684d822753f7b: Status 404 returned error can't find the container with id ce021deef11db514810c343a7c4f5b0d8082f60c8597c8b87ad684d822753f7b
Nov 02 10:19:05.276925 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:05.276826    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6" event={"ID":"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149","Type":"ContainerStarted","Data":"61c06da3d7b9969e11a904559610f25711e17955fd429ce317d621160a7a8089"}
Nov 02 10:19:05.276925 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:05.276886    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6" event={"ID":"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149","Type":"ContainerStarted","Data":"ce021deef11db514810c343a7c4f5b0d8082f60c8597c8b87ad684d822753f7b"}
Nov 02 10:19:05.299070 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:05.298971    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6" podStartSLOduration=1.29895191 podStartE2EDuration="1.29895191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:19:05.297870055 +0000 UTC m=+2622.236156843" watchObservedRunningTime="2026-11-02 10:19:05.29895191 +0000 UTC m=+2622.237238687"
Nov 02 10:19:06.283924 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:06.283888    4449 generic.go:356] "Generic (PLEG): container finished" podID="6a07b3dc-ccdf-41cc-88ca-c9d5475cf149" containerID="61c06da3d7b9969e11a904559610f25711e17955fd429ce317d621160a7a8089" exitCode=0
Nov 02 10:19:06.284366 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:06.283974    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6" event={"ID":"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149","Type":"ContainerDied","Data":"61c06da3d7b9969e11a904559610f25711e17955fd429ce317d621160a7a8089"}
Nov 02 10:19:06.546874 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:19:06.546834    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:19:07.424106 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:07.424084    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"
Nov 02 10:19:07.552266 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:07.552137    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-xks-ca\") pod \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\" (UID: \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\") "
Nov 02 10:19:07.552266 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:07.552187    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-openshift-ca\") pod \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\" (UID: \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\") "
Nov 02 10:19:07.552546 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:07.552363    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-kube-api-access-flnnv\" (UniqueName: \"kubernetes.io/projected/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-kube-api-access-flnnv\") pod \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\" (UID: \"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149\") "
Nov 02 10:19:07.552606 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:07.552575    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-openshift-ca" pod "6a07b3dc-ccdf-41cc-88ca-c9d5475cf149" (UID: "6a07b3dc-ccdf-41cc-88ca-c9d5475cf149"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:19:07.552727 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:07.552711    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:19:07.554510 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:07.554489    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-xks-ca" pod "6a07b3dc-ccdf-41cc-88ca-c9d5475cf149" (UID: "6a07b3dc-ccdf-41cc-88ca-c9d5475cf149"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:19:07.554633 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:07.554609    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-kube-api-access-flnnv" pod "6a07b3dc-ccdf-41cc-88ca-c9d5475cf149" (UID: "6a07b3dc-ccdf-41cc-88ca-c9d5475cf149"). InnerVolumeSpecName "kube-api-access-flnnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:19:07.653387 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:07.653340    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:19:07.653387 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:07.653377    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flnnv\" (UniqueName: \"kubernetes.io/projected/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149-kube-api-access-flnnv\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:19:08.297004 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:08.296970    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6" event={"ID":"6a07b3dc-ccdf-41cc-88ca-c9d5475cf149","Type":"ContainerDied","Data":"ce021deef11db514810c343a7c4f5b0d8082f60c8597c8b87ad684d822753f7b"}
Nov 02 10:19:08.297185 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:08.297018    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce021deef11db514810c343a7c4f5b0d8082f60c8597c8b87ad684d822753f7b"
Nov 02 10:19:08.297185 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:08.296979    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"
Nov 02 10:19:21.409015 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.408981    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"]
Nov 02 10:19:21.409775 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.409735    4449 cpu_manager.go:414] "containerMap: removing container" podUID="6a07b3dc-ccdf-41cc-88ca-c9d5475cf149" containerName="revoke-keys"
Nov 02 10:19:21.409890 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.409775    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a07b3dc-ccdf-41cc-88ca-c9d5475cf149" containerName="revoke-keys"
Nov 02 10:19:21.409960 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.409935    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a07b3dc-ccdf-41cc-88ca-c9d5475cf149" containerName="revoke-keys"
Nov 02 10:19:21.413691 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.413665    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"
Nov 02 10:19:21.419091 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.419036    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:19:21.419409 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.419075    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:19:21.419409 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.419099    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:19:21.419409 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.419170    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:19:21.423683 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.423653    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"]
Nov 02 10:19:21.540099 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.540055    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qcfvv\" (UID: \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"
Nov 02 10:19:21.540321 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.540108    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q4xm\" (UniqueName: \"kubernetes.io/projected/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-kube-api-access-7q4xm\") pod \"maas-api-revoke-sub-a4aef7d7-qcfvv\" (UID: \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"
Nov 02 10:19:21.540375 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.540324    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qcfvv\" (UID: \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"
Nov 02 10:19:21.641677 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.641634    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qcfvv\" (UID: \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"
Nov 02 10:19:21.641829 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.641702    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qcfvv\" (UID: \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"
Nov 02 10:19:21.641829 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.641729    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q4xm\" (UniqueName: \"kubernetes.io/projected/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-kube-api-access-7q4xm\") pod \"maas-api-revoke-sub-a4aef7d7-qcfvv\" (UID: \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"
Nov 02 10:19:21.642459 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.642437    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qcfvv\" (UID: \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"
Nov 02 10:19:21.644299 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.644281    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qcfvv\" (UID: \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"
Nov 02 10:19:21.654579 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.654553    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q4xm\" (UniqueName: \"kubernetes.io/projected/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-kube-api-access-7q4xm\") pod \"maas-api-revoke-sub-a4aef7d7-qcfvv\" (UID: \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"
Nov 02 10:19:21.734416 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:21.734326    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"
Nov 02 10:19:22.012009 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:22.011977    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"]
Nov 02 10:19:22.012144 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:19:22.012003    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1e62c5_3d95_4ebf_a9a0_1d840a002b82.slice/crio-627650e6bd9a70fd6079dff6670d85b02fe58c7550b18ca6bccadec3098e441b WatchSource:0}: Error finding container 627650e6bd9a70fd6079dff6670d85b02fe58c7550b18ca6bccadec3098e441b: Status 404 returned error can't find the container with id 627650e6bd9a70fd6079dff6670d85b02fe58c7550b18ca6bccadec3098e441b
Nov 02 10:19:22.421684 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:22.421646    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv" event={"ID":"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82","Type":"ContainerStarted","Data":"f6eb3a5640b5233f36c156a509640e2acbe9e155251eaadab3ffe5057bfbe7f3"}
Nov 02 10:19:22.422113 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:22.421706    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv" event={"ID":"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82","Type":"ContainerStarted","Data":"627650e6bd9a70fd6079dff6670d85b02fe58c7550b18ca6bccadec3098e441b"}
Nov 02 10:19:22.442677 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:22.442622    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv" podStartSLOduration=1.442608857 podStartE2EDuration="1.442608857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:19:22.440869007 +0000 UTC m=+2639.379155793" watchObservedRunningTime="2026-11-02 10:19:22.442608857 +0000 UTC m=+2639.380895632"
Nov 02 10:19:23.428772 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:23.428737    4449 generic.go:356] "Generic (PLEG): container finished" podID="aa1e62c5-3d95-4ebf-a9a0-1d840a002b82" containerID="f6eb3a5640b5233f36c156a509640e2acbe9e155251eaadab3ffe5057bfbe7f3" exitCode=0
Nov 02 10:19:23.429189 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:23.428791    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv" event={"ID":"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82","Type":"ContainerDied","Data":"f6eb3a5640b5233f36c156a509640e2acbe9e155251eaadab3ffe5057bfbe7f3"}
Nov 02 10:19:24.573785 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:24.573759    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"
Nov 02 10:19:24.670101 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:24.670068    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-kube-api-access-7q4xm\" (UniqueName: \"kubernetes.io/projected/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-kube-api-access-7q4xm\") pod \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\" (UID: \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\") "
Nov 02 10:19:24.670324 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:24.670158    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-xks-ca\") pod \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\" (UID: \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\") "
Nov 02 10:19:24.670324 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:24.670209    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-openshift-ca\") pod \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\" (UID: \"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82\") "
Nov 02 10:19:24.670632 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:24.670606    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-openshift-ca" pod "aa1e62c5-3d95-4ebf-a9a0-1d840a002b82" (UID: "aa1e62c5-3d95-4ebf-a9a0-1d840a002b82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:19:24.672495 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:24.672465    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-xks-ca" pod "aa1e62c5-3d95-4ebf-a9a0-1d840a002b82" (UID: "aa1e62c5-3d95-4ebf-a9a0-1d840a002b82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:19:24.672495 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:24.672482    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-kube-api-access-7q4xm" pod "aa1e62c5-3d95-4ebf-a9a0-1d840a002b82" (UID: "aa1e62c5-3d95-4ebf-a9a0-1d840a002b82"). InnerVolumeSpecName "kube-api-access-7q4xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:19:24.771853 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:24.771762    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q4xm\" (UniqueName: \"kubernetes.io/projected/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-kube-api-access-7q4xm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:19:24.771853 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:24.771798    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:19:24.771853 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:24.771809    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:19:25.441387 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:25.441355    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"
Nov 02 10:19:25.441387 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:25.441380    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv" event={"ID":"aa1e62c5-3d95-4ebf-a9a0-1d840a002b82","Type":"ContainerDied","Data":"627650e6bd9a70fd6079dff6670d85b02fe58c7550b18ca6bccadec3098e441b"}
Nov 02 10:19:25.441597 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:25.441423    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="627650e6bd9a70fd6079dff6670d85b02fe58c7550b18ca6bccadec3098e441b"
Nov 02 10:19:36.626617 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:19:36.626359    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:19:37.144363 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.144325    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"]
Nov 02 10:19:37.144801 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.144789    4449 cpu_manager.go:414] "containerMap: removing container" podUID="aa1e62c5-3d95-4ebf-a9a0-1d840a002b82" containerName="revoke-keys"
Nov 02 10:19:37.144848 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.144803    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa1e62c5-3d95-4ebf-a9a0-1d840a002b82" containerName="revoke-keys"
Nov 02 10:19:37.144885 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.144872    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa1e62c5-3d95-4ebf-a9a0-1d840a002b82" containerName="revoke-keys"
Nov 02 10:19:37.148237 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.148203    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"
Nov 02 10:19:37.154578 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.154552    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:19:37.172468 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.172428    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:19:37.172793 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.172771    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:19:37.199700 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.183142    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:19:37.199970 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.187011    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"]
Nov 02 10:19:37.236752 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.236719    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-skqqb\" (UID: \"de73dc84-9811-4f8c-b29b-9262e818398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"
Nov 02 10:19:37.237076 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.237050    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-skqqb\" (UID: \"de73dc84-9811-4f8c-b29b-9262e818398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"
Nov 02 10:19:37.237370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.237346    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzdhx\" (UniqueName: \"kubernetes.io/projected/de73dc84-9811-4f8c-b29b-9262e818398d-kube-api-access-hzdhx\") pod \"maas-api-revoke-sub-79805ca9-skqqb\" (UID: \"de73dc84-9811-4f8c-b29b-9262e818398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"
Nov 02 10:19:37.338560 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.338524    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-skqqb\" (UID: \"de73dc84-9811-4f8c-b29b-9262e818398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"
Nov 02 10:19:37.338788 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.338580    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-skqqb\" (UID: \"de73dc84-9811-4f8c-b29b-9262e818398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"
Nov 02 10:19:37.338788 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.338669    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzdhx\" (UniqueName: \"kubernetes.io/projected/de73dc84-9811-4f8c-b29b-9262e818398d-kube-api-access-hzdhx\") pod \"maas-api-revoke-sub-79805ca9-skqqb\" (UID: \"de73dc84-9811-4f8c-b29b-9262e818398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"
Nov 02 10:19:37.339303 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.339273    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-skqqb\" (UID: \"de73dc84-9811-4f8c-b29b-9262e818398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"
Nov 02 10:19:37.341293 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.341267    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-skqqb\" (UID: \"de73dc84-9811-4f8c-b29b-9262e818398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"
Nov 02 10:19:37.390967 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.390891    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzdhx\" (UniqueName: \"kubernetes.io/projected/de73dc84-9811-4f8c-b29b-9262e818398d-kube-api-access-hzdhx\") pod \"maas-api-revoke-sub-79805ca9-skqqb\" (UID: \"de73dc84-9811-4f8c-b29b-9262e818398d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"
Nov 02 10:19:37.538948 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.538917    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"
Nov 02 10:19:38.003300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:37.998466    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"]
Nov 02 10:19:38.004154 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:19:38.004093    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde73dc84_9811_4f8c_b29b_9262e818398d.slice/crio-a4b61cbc1de118818541ee96b3e2e8b1f3fb62d51e88ac99612c854f13767aaf WatchSource:0}: Error finding container a4b61cbc1de118818541ee96b3e2e8b1f3fb62d51e88ac99612c854f13767aaf: Status 404 returned error can't find the container with id a4b61cbc1de118818541ee96b3e2e8b1f3fb62d51e88ac99612c854f13767aaf
Nov 02 10:19:38.535669 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:38.535635    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb" event={"ID":"de73dc84-9811-4f8c-b29b-9262e818398d","Type":"ContainerStarted","Data":"a4b61cbc1de118818541ee96b3e2e8b1f3fb62d51e88ac99612c854f13767aaf"}
Nov 02 10:19:39.543094 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:39.543061    4449 generic.go:356] "Generic (PLEG): container finished" podID="de73dc84-9811-4f8c-b29b-9262e818398d" containerID="802b63f15a506e6763c79952161ac4a3f6c0422a00fc0fb6702dec61c289174b" exitCode=0
Nov 02 10:19:39.543511 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:39.543149    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb" event={"ID":"de73dc84-9811-4f8c-b29b-9262e818398d","Type":"ContainerDied","Data":"802b63f15a506e6763c79952161ac4a3f6c0422a00fc0fb6702dec61c289174b"}
Nov 02 10:19:40.689822 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:40.689794    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"
Nov 02 10:19:40.800885 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:40.800781    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-xks-ca\") pod \"de73dc84-9811-4f8c-b29b-9262e818398d\" (UID: \"de73dc84-9811-4f8c-b29b-9262e818398d\") "
Nov 02 10:19:40.800885 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:40.800879    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de73dc84-9811-4f8c-b29b-9262e818398d-kube-api-access-hzdhx\" (UniqueName: \"kubernetes.io/projected/de73dc84-9811-4f8c-b29b-9262e818398d-kube-api-access-hzdhx\") pod \"de73dc84-9811-4f8c-b29b-9262e818398d\" (UID: \"de73dc84-9811-4f8c-b29b-9262e818398d\") "
Nov 02 10:19:40.801127 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:40.800933    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-openshift-ca\") pod \"de73dc84-9811-4f8c-b29b-9262e818398d\" (UID: \"de73dc84-9811-4f8c-b29b-9262e818398d\") "
Nov 02 10:19:40.801480 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:40.801449    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-openshift-ca" pod "de73dc84-9811-4f8c-b29b-9262e818398d" (UID: "de73dc84-9811-4f8c-b29b-9262e818398d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:19:40.803022 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:40.802991    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-xks-ca" pod "de73dc84-9811-4f8c-b29b-9262e818398d" (UID: "de73dc84-9811-4f8c-b29b-9262e818398d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:19:40.803130 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:40.803098    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de73dc84-9811-4f8c-b29b-9262e818398d-kube-api-access-hzdhx" pod "de73dc84-9811-4f8c-b29b-9262e818398d" (UID: "de73dc84-9811-4f8c-b29b-9262e818398d"). InnerVolumeSpecName "kube-api-access-hzdhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:19:40.901856 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:40.901820    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:19:40.901856 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:40.901854    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzdhx\" (UniqueName: \"kubernetes.io/projected/de73dc84-9811-4f8c-b29b-9262e818398d-kube-api-access-hzdhx\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:19:40.901856 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:40.901865    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de73dc84-9811-4f8c-b29b-9262e818398d-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:19:41.556823 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:41.556790    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb" event={"ID":"de73dc84-9811-4f8c-b29b-9262e818398d","Type":"ContainerDied","Data":"a4b61cbc1de118818541ee96b3e2e8b1f3fb62d51e88ac99612c854f13767aaf"}
Nov 02 10:19:41.557006 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:41.556827    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"
Nov 02 10:19:41.557006 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:41.556840    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4b61cbc1de118818541ee96b3e2e8b1f3fb62d51e88ac99612c854f13767aaf"
Nov 02 10:19:51.786328 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.786284    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"]
Nov 02 10:19:51.787800 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.787769    4449 cpu_manager.go:414] "containerMap: removing container" podUID="de73dc84-9811-4f8c-b29b-9262e818398d" containerName="revoke-keys"
Nov 02 10:19:51.788827 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.788807    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de73dc84-9811-4f8c-b29b-9262e818398d" containerName="revoke-keys"
Nov 02 10:19:51.789381 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.789363    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="de73dc84-9811-4f8c-b29b-9262e818398d" containerName="revoke-keys"
Nov 02 10:19:51.807372 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.807341    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"
Nov 02 10:19:51.838321 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.838293    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:19:51.838741 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.838719    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:19:51.839498 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.839477    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:19:51.839723 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.839707    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:19:51.844925 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.844304    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:19:51.844925 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.844382    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:19:51.846930 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.846666    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"]
Nov 02 10:19:51.942929 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.942894    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-h2b7t\" (UID: \"47080f32-b404-4761-a197-8c5b9e7c013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"
Nov 02 10:19:51.943092 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.943019    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vd98\" (UniqueName: \"kubernetes.io/projected/47080f32-b404-4761-a197-8c5b9e7c013c-kube-api-access-4vd98\") pod \"maas-api-revoke-sub-0b7e8edb-h2b7t\" (UID: \"47080f32-b404-4761-a197-8c5b9e7c013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"
Nov 02 10:19:51.943092 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:51.943056    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-h2b7t\" (UID: \"47080f32-b404-4761-a197-8c5b9e7c013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"
Nov 02 10:19:52.044355 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:52.044260    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vd98\" (UniqueName: \"kubernetes.io/projected/47080f32-b404-4761-a197-8c5b9e7c013c-kube-api-access-4vd98\") pod \"maas-api-revoke-sub-0b7e8edb-h2b7t\" (UID: \"47080f32-b404-4761-a197-8c5b9e7c013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"
Nov 02 10:19:52.044355 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:52.044301    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-h2b7t\" (UID: \"47080f32-b404-4761-a197-8c5b9e7c013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"
Nov 02 10:19:52.044586 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:52.044381    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-h2b7t\" (UID: \"47080f32-b404-4761-a197-8c5b9e7c013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"
Nov 02 10:19:52.045341 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:52.045318    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-h2b7t\" (UID: \"47080f32-b404-4761-a197-8c5b9e7c013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"
Nov 02 10:19:52.046817 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:52.046786    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-h2b7t\" (UID: \"47080f32-b404-4761-a197-8c5b9e7c013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"
Nov 02 10:19:52.104240 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:52.104011    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vd98\" (UniqueName: \"kubernetes.io/projected/47080f32-b404-4761-a197-8c5b9e7c013c-kube-api-access-4vd98\") pod \"maas-api-revoke-sub-0b7e8edb-h2b7t\" (UID: \"47080f32-b404-4761-a197-8c5b9e7c013c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"
Nov 02 10:19:52.236959 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:52.236925    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"
Nov 02 10:19:52.716297 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:52.697958    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"]
Nov 02 10:19:52.723107 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:19:52.723062    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47080f32_b404_4761_a197_8c5b9e7c013c.slice/crio-c72665d640ba4b1af714ae45c0574a0bb1771d6ac4a4180d0126e2568c3f4e8c WatchSource:0}: Error finding container c72665d640ba4b1af714ae45c0574a0bb1771d6ac4a4180d0126e2568c3f4e8c: Status 404 returned error can't find the container with id c72665d640ba4b1af714ae45c0574a0bb1771d6ac4a4180d0126e2568c3f4e8c
Nov 02 10:19:53.644596 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:53.644559    4449 generic.go:356] "Generic (PLEG): container finished" podID="47080f32-b404-4761-a197-8c5b9e7c013c" containerID="dfbc2a06267d290f1d6516073a357f0b3bf394250749333d461c505aec172196" exitCode=0
Nov 02 10:19:53.644991 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:53.644644    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t" event={"ID":"47080f32-b404-4761-a197-8c5b9e7c013c","Type":"ContainerDied","Data":"dfbc2a06267d290f1d6516073a357f0b3bf394250749333d461c505aec172196"}
Nov 02 10:19:53.644991 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:53.644692    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t" event={"ID":"47080f32-b404-4761-a197-8c5b9e7c013c","Type":"ContainerStarted","Data":"c72665d640ba4b1af714ae45c0574a0bb1771d6ac4a4180d0126e2568c3f4e8c"}
Nov 02 10:19:54.793690 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:54.793665    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"
Nov 02 10:19:54.924258 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:54.924157    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-openshift-ca\") pod \"47080f32-b404-4761-a197-8c5b9e7c013c\" (UID: \"47080f32-b404-4761-a197-8c5b9e7c013c\") "
Nov 02 10:19:54.924447 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:54.924286    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-xks-ca\") pod \"47080f32-b404-4761-a197-8c5b9e7c013c\" (UID: \"47080f32-b404-4761-a197-8c5b9e7c013c\") "
Nov 02 10:19:54.924447 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:54.924328    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47080f32-b404-4761-a197-8c5b9e7c013c-kube-api-access-4vd98\" (UniqueName: \"kubernetes.io/projected/47080f32-b404-4761-a197-8c5b9e7c013c-kube-api-access-4vd98\") pod \"47080f32-b404-4761-a197-8c5b9e7c013c\" (UID: \"47080f32-b404-4761-a197-8c5b9e7c013c\") "
Nov 02 10:19:54.924620 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:54.924589    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-openshift-ca" pod "47080f32-b404-4761-a197-8c5b9e7c013c" (UID: "47080f32-b404-4761-a197-8c5b9e7c013c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:19:54.926735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:54.926699    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-xks-ca" pod "47080f32-b404-4761-a197-8c5b9e7c013c" (UID: "47080f32-b404-4761-a197-8c5b9e7c013c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:19:54.926853 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:54.926757    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47080f32-b404-4761-a197-8c5b9e7c013c-kube-api-access-4vd98" pod "47080f32-b404-4761-a197-8c5b9e7c013c" (UID: "47080f32-b404-4761-a197-8c5b9e7c013c"). InnerVolumeSpecName "kube-api-access-4vd98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:19:55.025095 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:55.025058    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:19:55.025095 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:55.025090    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vd98\" (UniqueName: \"kubernetes.io/projected/47080f32-b404-4761-a197-8c5b9e7c013c-kube-api-access-4vd98\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:19:55.025095 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:55.025099    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47080f32-b404-4761-a197-8c5b9e7c013c-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:19:55.658790 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:55.658754    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t" event={"ID":"47080f32-b404-4761-a197-8c5b9e7c013c","Type":"ContainerDied","Data":"c72665d640ba4b1af714ae45c0574a0bb1771d6ac4a4180d0126e2568c3f4e8c"}
Nov 02 10:19:55.658790 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:55.658789    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"
Nov 02 10:19:55.659007 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:19:55.658828    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c72665d640ba4b1af714ae45c0574a0bb1771d6ac4a4180d0126e2568c3f4e8c"
Nov 02 10:20:02.425588 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.425551    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"]
Nov 02 10:20:02.426064 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.426049    4449 cpu_manager.go:414] "containerMap: removing container" podUID="47080f32-b404-4761-a197-8c5b9e7c013c" containerName="revoke-keys"
Nov 02 10:20:02.426130 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.426065    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47080f32-b404-4761-a197-8c5b9e7c013c" containerName="revoke-keys"
Nov 02 10:20:02.426178 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.426153    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="47080f32-b404-4761-a197-8c5b9e7c013c" containerName="revoke-keys"
Nov 02 10:20:02.429564 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.429540    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"
Nov 02 10:20:02.434539 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.434488    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:20:02.435794 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.435772    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:20:02.439641 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.439333    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:20:02.439641 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.439507    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:20:02.442741 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.442716    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"]
Nov 02 10:20:02.605395 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.605363    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s525g\" (UID: \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"
Nov 02 10:20:02.605582 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.605441    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flzd6\" (UniqueName: \"kubernetes.io/projected/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-kube-api-access-flzd6\") pod \"maas-api-revoke-sub-3d4156f1-s525g\" (UID: \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"
Nov 02 10:20:02.605582 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.605557    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s525g\" (UID: \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"
Nov 02 10:20:02.706321 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.706234    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s525g\" (UID: \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"
Nov 02 10:20:02.706321 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.706286    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s525g\" (UID: \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"
Nov 02 10:20:02.706572 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.706355    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flzd6\" (UniqueName: \"kubernetes.io/projected/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-kube-api-access-flzd6\") pod \"maas-api-revoke-sub-3d4156f1-s525g\" (UID: \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"
Nov 02 10:20:02.706975 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.706946    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s525g\" (UID: \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"
Nov 02 10:20:02.708827 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.708804    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s525g\" (UID: \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"
Nov 02 10:20:02.719031 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.719009    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flzd6\" (UniqueName: \"kubernetes.io/projected/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-kube-api-access-flzd6\") pod \"maas-api-revoke-sub-3d4156f1-s525g\" (UID: \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"
Nov 02 10:20:02.747169 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:02.747140    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"
Nov 02 10:20:03.025912 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:03.025885    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"]
Nov 02 10:20:03.027087 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:20:03.027057    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8944d99a_856d_4fe6_a2fe_d12ac3f994fe.slice/crio-4b11ba571f0e11c1ae5f063669088276d3387a083ccceddf088906366193987f WatchSource:0}: Error finding container 4b11ba571f0e11c1ae5f063669088276d3387a083ccceddf088906366193987f: Status 404 returned error can't find the container with id 4b11ba571f0e11c1ae5f063669088276d3387a083ccceddf088906366193987f
Nov 02 10:20:03.706917 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:03.706810    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g" event={"ID":"8944d99a-856d-4fe6-a2fe-d12ac3f994fe","Type":"ContainerStarted","Data":"8e3f3e3d7aa16553bc9f3a81c75b01becde340920eb629d7430a312e35a8da62"}
Nov 02 10:20:03.706917 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:03.706871    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g" event={"ID":"8944d99a-856d-4fe6-a2fe-d12ac3f994fe","Type":"ContainerStarted","Data":"4b11ba571f0e11c1ae5f063669088276d3387a083ccceddf088906366193987f"}
Nov 02 10:20:03.727452 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:03.727398    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g" podStartSLOduration=1.727382201 podStartE2EDuration="1.727382201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:20:03.725601189 +0000 UTC m=+2680.663887987" watchObservedRunningTime="2026-11-02 10:20:03.727382201 +0000 UTC m=+2680.665668978"
Nov 02 10:20:04.714612 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:04.714580    4449 generic.go:356] "Generic (PLEG): container finished" podID="8944d99a-856d-4fe6-a2fe-d12ac3f994fe" containerID="8e3f3e3d7aa16553bc9f3a81c75b01becde340920eb629d7430a312e35a8da62" exitCode=0
Nov 02 10:20:04.714996 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:04.714664    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g" event={"ID":"8944d99a-856d-4fe6-a2fe-d12ac3f994fe","Type":"ContainerDied","Data":"8e3f3e3d7aa16553bc9f3a81c75b01becde340920eb629d7430a312e35a8da62"}
Nov 02 10:20:05.860619 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:05.860595    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"
Nov 02 10:20:05.940445 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:05.940395    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-openshift-ca\") pod \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\" (UID: \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\") "
Nov 02 10:20:05.940674 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:05.940517    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-kube-api-access-flzd6\" (UniqueName: \"kubernetes.io/projected/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-kube-api-access-flzd6\") pod \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\" (UID: \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\") "
Nov 02 10:20:05.940674 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:05.940621    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-xks-ca\") pod \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\" (UID: \"8944d99a-856d-4fe6-a2fe-d12ac3f994fe\") "
Nov 02 10:20:05.940922 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:05.940895    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-openshift-ca" pod "8944d99a-856d-4fe6-a2fe-d12ac3f994fe" (UID: "8944d99a-856d-4fe6-a2fe-d12ac3f994fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:20:05.942767 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:05.942738    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-xks-ca" pod "8944d99a-856d-4fe6-a2fe-d12ac3f994fe" (UID: "8944d99a-856d-4fe6-a2fe-d12ac3f994fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:20:05.942896 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:05.942873    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-kube-api-access-flzd6" pod "8944d99a-856d-4fe6-a2fe-d12ac3f994fe" (UID: "8944d99a-856d-4fe6-a2fe-d12ac3f994fe"). InnerVolumeSpecName "kube-api-access-flzd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:20:06.041362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:06.041334    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:20:06.041362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:06.041362    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flzd6\" (UniqueName: \"kubernetes.io/projected/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-kube-api-access-flzd6\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:20:06.041549 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:06.041374    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8944d99a-856d-4fe6-a2fe-d12ac3f994fe-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:20:06.547141 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:20:06.547106    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:20:06.728871 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:06.728842    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"
Nov 02 10:20:06.729071 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:06.728883    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g" event={"ID":"8944d99a-856d-4fe6-a2fe-d12ac3f994fe","Type":"ContainerDied","Data":"4b11ba571f0e11c1ae5f063669088276d3387a083ccceddf088906366193987f"}
Nov 02 10:20:06.729071 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:06.728938    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b11ba571f0e11c1ae5f063669088276d3387a083ccceddf088906366193987f"
Nov 02 10:20:17.942242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:17.941793    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"]
Nov 02 10:20:17.942242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:17.941861    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s525g"]
Nov 02 10:20:17.958676 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:17.958631    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"]
Nov 02 10:20:17.959420 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:17.959394    4449 cpu_manager.go:414] "containerMap: removing container" podUID="8944d99a-856d-4fe6-a2fe-d12ac3f994fe" containerName="revoke-keys"
Nov 02 10:20:17.959420 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:17.959419    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8944d99a-856d-4fe6-a2fe-d12ac3f994fe" containerName="revoke-keys"
Nov 02 10:20:17.959627 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:17.959578    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="8944d99a-856d-4fe6-a2fe-d12ac3f994fe" containerName="revoke-keys"
Nov 02 10:20:17.963659 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:17.963631    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"
Nov 02 10:20:17.969837 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:17.969809    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:20:17.970321 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:17.970294    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:20:17.970551 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:17.970427    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:20:17.970678 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:17.970535    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:20:17.985815 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:17.985749    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"]
Nov 02 10:20:18.057741 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:18.057699    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7p84\" (UniqueName: \"kubernetes.io/projected/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-kube-api-access-p7p84\") pod \"maas-api-revoke-sub-3d4156f1-jzmhq\" (UID: \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"
Nov 02 10:20:18.057942 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:18.057828    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jzmhq\" (UID: \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"
Nov 02 10:20:18.057942 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:18.057864    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jzmhq\" (UID: \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"
Nov 02 10:20:18.159093 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:18.159059    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jzmhq\" (UID: \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"
Nov 02 10:20:18.159370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:18.159113    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jzmhq\" (UID: \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"
Nov 02 10:20:18.159370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:18.159327    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7p84\" (UniqueName: \"kubernetes.io/projected/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-kube-api-access-p7p84\") pod \"maas-api-revoke-sub-3d4156f1-jzmhq\" (UID: \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"
Nov 02 10:20:18.159926 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:18.159891    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jzmhq\" (UID: \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"
Nov 02 10:20:18.162561 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:18.162531    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jzmhq\" (UID: \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"
Nov 02 10:20:18.172399 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:18.172371    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7p84\" (UniqueName: \"kubernetes.io/projected/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-kube-api-access-p7p84\") pod \"maas-api-revoke-sub-3d4156f1-jzmhq\" (UID: \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"
Nov 02 10:20:18.288862 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:18.288817    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"
Nov 02 10:20:18.616613 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:18.608378    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"]
Nov 02 10:20:18.851047 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:18.850964    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq" event={"ID":"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d","Type":"ContainerStarted","Data":"d126b788bed34f7d89b6e3cebaba90c0fa7b26e9fb6cc2ff0372321818553cf4"}
Nov 02 10:20:19.225666 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:19.225627    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8944d99a-856d-4fe6-a2fe-d12ac3f994fe" path="/var/lib/kubelet/pods/8944d99a-856d-4fe6-a2fe-d12ac3f994fe/volumes"
Nov 02 10:20:19.886362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:19.886304    4449 generic.go:356] "Generic (PLEG): container finished" podID="0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d" containerID="cc54b965c8d1147315147603b01f80b5c48b06b6c43f12ed3cf2ab3e9bcc6345" exitCode=0
Nov 02 10:20:19.886807 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:19.886778    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq" event={"ID":"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d","Type":"ContainerDied","Data":"cc54b965c8d1147315147603b01f80b5c48b06b6c43f12ed3cf2ab3e9bcc6345"}
Nov 02 10:20:21.076405 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:21.076372    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"
Nov 02 10:20:21.127577 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:21.127540    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-kube-api-access-p7p84\" (UniqueName: \"kubernetes.io/projected/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-kube-api-access-p7p84\") pod \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\" (UID: \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\") "
Nov 02 10:20:21.127775 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:21.127597    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-openshift-ca\") pod \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\" (UID: \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\") "
Nov 02 10:20:21.127775 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:21.127634    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-xks-ca\") pod \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\" (UID: \"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d\") "
Nov 02 10:20:21.128411 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:21.128350    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-openshift-ca" pod "0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d" (UID: "0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:20:21.130577 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:21.130533    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-kube-api-access-p7p84" pod "0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d" (UID: "0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d"). InnerVolumeSpecName "kube-api-access-p7p84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:20:21.136663 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:21.136630    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-xks-ca" pod "0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d" (UID: "0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:20:21.229012 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:21.228822    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7p84\" (UniqueName: \"kubernetes.io/projected/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-kube-api-access-p7p84\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:20:21.229012 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:21.228854    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:20:21.229012 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:21.228870    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:20:21.905017 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:21.904983    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"
Nov 02 10:20:21.905211 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:21.905015    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq" event={"ID":"0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d","Type":"ContainerDied","Data":"d126b788bed34f7d89b6e3cebaba90c0fa7b26e9fb6cc2ff0372321818553cf4"}
Nov 02 10:20:21.905211 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:21.905064    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d126b788bed34f7d89b6e3cebaba90c0fa7b26e9fb6cc2ff0372321818553cf4"
Nov 02 10:20:36.544959 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:20:36.544915    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:20:41.614330 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.614283    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"]
Nov 02 10:20:41.614760 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.614738    4449 cpu_manager.go:414] "containerMap: removing container" podUID="0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d" containerName="revoke-keys"
Nov 02 10:20:41.614760 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.614749    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d" containerName="revoke-keys"
Nov 02 10:20:41.614836 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.614824    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d" containerName="revoke-keys"
Nov 02 10:20:41.619329 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.619303    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"
Nov 02 10:20:41.624052 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.623745    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:20:41.624924 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.624891    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:20:41.625069 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.624959    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:20:41.625600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.625577    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:20:41.629934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.629910    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"]
Nov 02 10:20:41.660202 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.660159    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bx4kv\" (UID: \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"
Nov 02 10:20:41.660430 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.660382    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxrch\" (UniqueName: \"kubernetes.io/projected/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-kube-api-access-sxrch\") pod \"maas-api-revoke-sub-e2fc9c5b-bx4kv\" (UID: \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"
Nov 02 10:20:41.660430 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.660423    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bx4kv\" (UID: \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"
Nov 02 10:20:41.761955 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.761910    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxrch\" (UniqueName: \"kubernetes.io/projected/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-kube-api-access-sxrch\") pod \"maas-api-revoke-sub-e2fc9c5b-bx4kv\" (UID: \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"
Nov 02 10:20:41.761955 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.761960    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bx4kv\" (UID: \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"
Nov 02 10:20:41.762196 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.762032    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bx4kv\" (UID: \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"
Nov 02 10:20:41.762737 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.762705    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bx4kv\" (UID: \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"
Nov 02 10:20:41.764563 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.764529    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-bx4kv\" (UID: \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"
Nov 02 10:20:41.775386 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.775365    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxrch\" (UniqueName: \"kubernetes.io/projected/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-kube-api-access-sxrch\") pod \"maas-api-revoke-sub-e2fc9c5b-bx4kv\" (UID: \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"
Nov 02 10:20:41.935389 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:41.935315    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"
Nov 02 10:20:42.222194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:42.222166    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"]
Nov 02 10:20:42.224350 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:20:42.224323    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8a7cb9_325d_4203_8bac_a2d88fdb9b31.slice/crio-f396eba8c56c824c141a05fa294f846a9890971734b9536388d4f7cf2922c05e WatchSource:0}: Error finding container f396eba8c56c824c141a05fa294f846a9890971734b9536388d4f7cf2922c05e: Status 404 returned error can't find the container with id f396eba8c56c824c141a05fa294f846a9890971734b9536388d4f7cf2922c05e
Nov 02 10:20:43.060555 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:43.060521    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv" event={"ID":"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31","Type":"ContainerStarted","Data":"6620970cf4a24cd58d77ea5e3bfc6d92b32dba25896004b74ede80837de93f2d"}
Nov 02 10:20:43.060994 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:43.060575    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv" event={"ID":"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31","Type":"ContainerStarted","Data":"f396eba8c56c824c141a05fa294f846a9890971734b9536388d4f7cf2922c05e"}
Nov 02 10:20:43.081767 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:43.081718    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv" podStartSLOduration=2.081704245 podStartE2EDuration="2.081704245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:20:43.078987916 +0000 UTC m=+2720.017274719" watchObservedRunningTime="2026-11-02 10:20:43.081704245 +0000 UTC m=+2720.019991022"
Nov 02 10:20:44.067496 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:44.067461    4449 generic.go:356] "Generic (PLEG): container finished" podID="8b8a7cb9-325d-4203-8bac-a2d88fdb9b31" containerID="6620970cf4a24cd58d77ea5e3bfc6d92b32dba25896004b74ede80837de93f2d" exitCode=0
Nov 02 10:20:44.067971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:44.067549    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv" event={"ID":"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31","Type":"ContainerDied","Data":"6620970cf4a24cd58d77ea5e3bfc6d92b32dba25896004b74ede80837de93f2d"}
Nov 02 10:20:45.214177 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:45.214150    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"
Nov 02 10:20:45.298606 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:45.298564    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-kube-api-access-sxrch\" (UniqueName: \"kubernetes.io/projected/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-kube-api-access-sxrch\") pod \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\" (UID: \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\") "
Nov 02 10:20:45.298812 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:45.298697    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-openshift-ca\") pod \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\" (UID: \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\") "
Nov 02 10:20:45.298888 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:45.298813    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-xks-ca\") pod \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\" (UID: \"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31\") "
Nov 02 10:20:45.299062 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:45.299036    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-openshift-ca" pod "8b8a7cb9-325d-4203-8bac-a2d88fdb9b31" (UID: "8b8a7cb9-325d-4203-8bac-a2d88fdb9b31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:20:45.299387 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:45.299321    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:20:45.301519 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:45.301497    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-xks-ca" pod "8b8a7cb9-325d-4203-8bac-a2d88fdb9b31" (UID: "8b8a7cb9-325d-4203-8bac-a2d88fdb9b31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:20:45.301616 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:45.301497    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-kube-api-access-sxrch" pod "8b8a7cb9-325d-4203-8bac-a2d88fdb9b31" (UID: "8b8a7cb9-325d-4203-8bac-a2d88fdb9b31"). InnerVolumeSpecName "kube-api-access-sxrch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:20:45.400179 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:45.400088    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:20:45.400179 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:45.400122    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxrch\" (UniqueName: \"kubernetes.io/projected/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31-kube-api-access-sxrch\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:20:46.080445 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:46.080415    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"
Nov 02 10:20:46.080660 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:46.080453    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv" event={"ID":"8b8a7cb9-325d-4203-8bac-a2d88fdb9b31","Type":"ContainerDied","Data":"f396eba8c56c824c141a05fa294f846a9890971734b9536388d4f7cf2922c05e"}
Nov 02 10:20:46.080660 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:46.080522    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f396eba8c56c824c141a05fa294f846a9890971734b9536388d4f7cf2922c05e"
Nov 02 10:20:51.839103 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:51.839066    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"]
Nov 02 10:20:51.839724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:51.839557    4449 cpu_manager.go:414] "containerMap: removing container" podUID="8b8a7cb9-325d-4203-8bac-a2d88fdb9b31" containerName="revoke-keys"
Nov 02 10:20:51.839724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:51.839569    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8a7cb9-325d-4203-8bac-a2d88fdb9b31" containerName="revoke-keys"
Nov 02 10:20:51.839724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:51.839660    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8a7cb9-325d-4203-8bac-a2d88fdb9b31" containerName="revoke-keys"
Nov 02 10:20:51.842837 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:51.842812    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"
Nov 02 10:20:51.853160 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:51.852063    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:20:51.853160 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:51.852131    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:20:51.853160 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:51.852135    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:20:51.853160 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:51.852081    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:20:51.853553 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:51.853526    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"]
Nov 02 10:20:51.968878 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:51.968837    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcl8h\" (UniqueName: \"kubernetes.io/projected/02a90313-de8e-42e0-aeba-24037f530cc5-kube-api-access-lcl8h\") pod \"maas-api-revoke-sub-5a29efb3-4spmr\" (UID: \"02a90313-de8e-42e0-aeba-24037f530cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"
Nov 02 10:20:51.968878 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:51.968883    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4spmr\" (UID: \"02a90313-de8e-42e0-aeba-24037f530cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"
Nov 02 10:20:51.969103 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:51.969035    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4spmr\" (UID: \"02a90313-de8e-42e0-aeba-24037f530cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"
Nov 02 10:20:52.070606 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:52.070571    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcl8h\" (UniqueName: \"kubernetes.io/projected/02a90313-de8e-42e0-aeba-24037f530cc5-kube-api-access-lcl8h\") pod \"maas-api-revoke-sub-5a29efb3-4spmr\" (UID: \"02a90313-de8e-42e0-aeba-24037f530cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"
Nov 02 10:20:52.070606 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:52.070611    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4spmr\" (UID: \"02a90313-de8e-42e0-aeba-24037f530cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"
Nov 02 10:20:52.070817 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:52.070672    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4spmr\" (UID: \"02a90313-de8e-42e0-aeba-24037f530cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"
Nov 02 10:20:52.071284 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:52.071265    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4spmr\" (UID: \"02a90313-de8e-42e0-aeba-24037f530cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"
Nov 02 10:20:52.073208 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:52.073188    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4spmr\" (UID: \"02a90313-de8e-42e0-aeba-24037f530cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"
Nov 02 10:20:52.097252 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:52.097162    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcl8h\" (UniqueName: \"kubernetes.io/projected/02a90313-de8e-42e0-aeba-24037f530cc5-kube-api-access-lcl8h\") pod \"maas-api-revoke-sub-5a29efb3-4spmr\" (UID: \"02a90313-de8e-42e0-aeba-24037f530cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"
Nov 02 10:20:52.159595 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:52.159558    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"
Nov 02 10:20:52.438143 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:52.438113    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"]
Nov 02 10:20:52.439110 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:20:52.439083    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a90313_de8e_42e0_aeba_24037f530cc5.slice/crio-88ed24fad1dfa2ec596aeb8f9e55e006041459618c1fbe21149ac13884b80486 WatchSource:0}: Error finding container 88ed24fad1dfa2ec596aeb8f9e55e006041459618c1fbe21149ac13884b80486: Status 404 returned error can't find the container with id 88ed24fad1dfa2ec596aeb8f9e55e006041459618c1fbe21149ac13884b80486
Nov 02 10:20:53.126500 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:53.126457    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr" event={"ID":"02a90313-de8e-42e0-aeba-24037f530cc5","Type":"ContainerStarted","Data":"f6d455246ca79a2ac9816d60ae6568186bad3e720f269a1c66a9cbc2411a5b6e"}
Nov 02 10:20:53.126901 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:53.126529    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr" event={"ID":"02a90313-de8e-42e0-aeba-24037f530cc5","Type":"ContainerStarted","Data":"88ed24fad1dfa2ec596aeb8f9e55e006041459618c1fbe21149ac13884b80486"}
Nov 02 10:20:53.147656 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:53.147590    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr" podStartSLOduration=2.14756972 podStartE2EDuration="2.14756972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:20:53.145456833 +0000 UTC m=+2730.083743624" watchObservedRunningTime="2026-11-02 10:20:53.14756972 +0000 UTC m=+2730.085856496"
Nov 02 10:20:54.144394 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:54.144349    4449 generic.go:356] "Generic (PLEG): container finished" podID="02a90313-de8e-42e0-aeba-24037f530cc5" containerID="f6d455246ca79a2ac9816d60ae6568186bad3e720f269a1c66a9cbc2411a5b6e" exitCode=0
Nov 02 10:20:54.144919 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:54.144426    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr" event={"ID":"02a90313-de8e-42e0-aeba-24037f530cc5","Type":"ContainerDied","Data":"f6d455246ca79a2ac9816d60ae6568186bad3e720f269a1c66a9cbc2411a5b6e"}
Nov 02 10:20:55.597251 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:55.597199    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"
Nov 02 10:20:55.736047 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:55.736007    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-xks-ca\") pod \"02a90313-de8e-42e0-aeba-24037f530cc5\" (UID: \"02a90313-de8e-42e0-aeba-24037f530cc5\") "
Nov 02 10:20:55.736259 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:55.736056    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a90313-de8e-42e0-aeba-24037f530cc5-kube-api-access-lcl8h\" (UniqueName: \"kubernetes.io/projected/02a90313-de8e-42e0-aeba-24037f530cc5-kube-api-access-lcl8h\") pod \"02a90313-de8e-42e0-aeba-24037f530cc5\" (UID: \"02a90313-de8e-42e0-aeba-24037f530cc5\") "
Nov 02 10:20:55.736259 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:55.736201    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-openshift-ca\") pod \"02a90313-de8e-42e0-aeba-24037f530cc5\" (UID: \"02a90313-de8e-42e0-aeba-24037f530cc5\") "
Nov 02 10:20:55.736592 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:55.736564    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-openshift-ca" pod "02a90313-de8e-42e0-aeba-24037f530cc5" (UID: "02a90313-de8e-42e0-aeba-24037f530cc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:20:55.738440 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:55.738410    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-xks-ca" pod "02a90313-de8e-42e0-aeba-24037f530cc5" (UID: "02a90313-de8e-42e0-aeba-24037f530cc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:20:55.738515 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:55.738438    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a90313-de8e-42e0-aeba-24037f530cc5-kube-api-access-lcl8h" pod "02a90313-de8e-42e0-aeba-24037f530cc5" (UID: "02a90313-de8e-42e0-aeba-24037f530cc5"). InnerVolumeSpecName "kube-api-access-lcl8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:20:55.837618 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:55.837582    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:20:55.837618 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:55.837609    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcl8h\" (UniqueName: \"kubernetes.io/projected/02a90313-de8e-42e0-aeba-24037f530cc5-kube-api-access-lcl8h\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:20:55.837618 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:55.837621    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02a90313-de8e-42e0-aeba-24037f530cc5-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:20:56.207295 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:56.206720    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr" event={"ID":"02a90313-de8e-42e0-aeba-24037f530cc5","Type":"ContainerDied","Data":"88ed24fad1dfa2ec596aeb8f9e55e006041459618c1fbe21149ac13884b80486"}
Nov 02 10:20:56.207295 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:56.206785    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88ed24fad1dfa2ec596aeb8f9e55e006041459618c1fbe21149ac13884b80486"
Nov 02 10:20:56.207295 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:20:56.206854    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"
Nov 02 10:21:06.548516 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:21:06.548425    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:21:07.989078 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:07.989011    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"]
Nov 02 10:21:07.991850 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:07.991825    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"]
Nov 02 10:21:07.992292 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:07.992279    4449 cpu_manager.go:414] "containerMap: removing container" podUID="02a90313-de8e-42e0-aeba-24037f530cc5" containerName="revoke-keys"
Nov 02 10:21:07.992330 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:07.992293    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a90313-de8e-42e0-aeba-24037f530cc5" containerName="revoke-keys"
Nov 02 10:21:07.992399 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:07.992389    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a90313-de8e-42e0-aeba-24037f530cc5" containerName="revoke-keys"
Nov 02 10:21:07.995786 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:07.995770    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"
Nov 02 10:21:08.017128 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.017094    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:21:08.017330 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.017182    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:21:08.017330 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.017186    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:21:08.017449 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.017354    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:21:08.017836 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.017803    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jzmhq"]
Nov 02 10:21:08.022095 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.022070    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"]
Nov 02 10:21:08.061148 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.061109    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56984b22-0667-4b89-aded-255238682def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jj42t\" (UID: \"56984b22-0667-4b89-aded-255238682def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"
Nov 02 10:21:08.061322 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.061202    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56984b22-0667-4b89-aded-255238682def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jj42t\" (UID: \"56984b22-0667-4b89-aded-255238682def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"
Nov 02 10:21:08.061448 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.061422    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t77d\" (UniqueName: \"kubernetes.io/projected/56984b22-0667-4b89-aded-255238682def-kube-api-access-9t77d\") pod \"maas-api-revoke-sub-3d4156f1-jj42t\" (UID: \"56984b22-0667-4b89-aded-255238682def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"
Nov 02 10:21:08.162993 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.162954    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56984b22-0667-4b89-aded-255238682def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jj42t\" (UID: \"56984b22-0667-4b89-aded-255238682def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"
Nov 02 10:21:08.163200 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.163007    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56984b22-0667-4b89-aded-255238682def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jj42t\" (UID: \"56984b22-0667-4b89-aded-255238682def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"
Nov 02 10:21:08.163200 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.163083    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t77d\" (UniqueName: \"kubernetes.io/projected/56984b22-0667-4b89-aded-255238682def-kube-api-access-9t77d\") pod \"maas-api-revoke-sub-3d4156f1-jj42t\" (UID: \"56984b22-0667-4b89-aded-255238682def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"
Nov 02 10:21:08.163803 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.163773    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56984b22-0667-4b89-aded-255238682def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jj42t\" (UID: \"56984b22-0667-4b89-aded-255238682def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"
Nov 02 10:21:08.165539 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.165515    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56984b22-0667-4b89-aded-255238682def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jj42t\" (UID: \"56984b22-0667-4b89-aded-255238682def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"
Nov 02 10:21:08.178004 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.177979    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t77d\" (UniqueName: \"kubernetes.io/projected/56984b22-0667-4b89-aded-255238682def-kube-api-access-9t77d\") pod \"maas-api-revoke-sub-3d4156f1-jj42t\" (UID: \"56984b22-0667-4b89-aded-255238682def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"
Nov 02 10:21:08.308680 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.308649    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"
Nov 02 10:21:08.596593 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:08.596569    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"]
Nov 02 10:21:08.598032 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:21:08.597997    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56984b22_0667_4b89_aded_255238682def.slice/crio-1942ec4d9efdbba1540416b95cf2f66e6f36e19504d9c2e559189e475c1e5db6 WatchSource:0}: Error finding container 1942ec4d9efdbba1540416b95cf2f66e6f36e19504d9c2e559189e475c1e5db6: Status 404 returned error can't find the container with id 1942ec4d9efdbba1540416b95cf2f66e6f36e19504d9c2e559189e475c1e5db6
Nov 02 10:21:09.205906 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:09.205827    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d" path="/var/lib/kubelet/pods/0f764c1c-9fcc-4a4c-aa54-e1a5e6f72b2d/volumes"
Nov 02 10:21:09.285271 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:09.285227    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t" event={"ID":"56984b22-0667-4b89-aded-255238682def","Type":"ContainerStarted","Data":"cf8df310c594f5f9c2138c66a5bf834b0885b7f54f8cbe472b85eb828a359e0e"}
Nov 02 10:21:09.285444 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:09.285287    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t" event={"ID":"56984b22-0667-4b89-aded-255238682def","Type":"ContainerStarted","Data":"1942ec4d9efdbba1540416b95cf2f66e6f36e19504d9c2e559189e475c1e5db6"}
Nov 02 10:21:09.309598 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:09.309545    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t" podStartSLOduration=2.309531465 podStartE2EDuration="2.309531465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:21:09.306170798 +0000 UTC m=+2746.244457595" watchObservedRunningTime="2026-11-02 10:21:09.309531465 +0000 UTC m=+2746.247818241"
Nov 02 10:21:10.293046 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:10.293012    4449 generic.go:356] "Generic (PLEG): container finished" podID="56984b22-0667-4b89-aded-255238682def" containerID="cf8df310c594f5f9c2138c66a5bf834b0885b7f54f8cbe472b85eb828a359e0e" exitCode=0
Nov 02 10:21:10.293497 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:10.293096    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t" event={"ID":"56984b22-0667-4b89-aded-255238682def","Type":"ContainerDied","Data":"cf8df310c594f5f9c2138c66a5bf834b0885b7f54f8cbe472b85eb828a359e0e"}
Nov 02 10:21:11.429073 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:11.429047    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"
Nov 02 10:21:11.496627 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:11.496591    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56984b22-0667-4b89-aded-255238682def-kube-api-access-9t77d\" (UniqueName: \"kubernetes.io/projected/56984b22-0667-4b89-aded-255238682def-kube-api-access-9t77d\") pod \"56984b22-0667-4b89-aded-255238682def\" (UID: \"56984b22-0667-4b89-aded-255238682def\") "
Nov 02 10:21:11.496826 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:11.496651    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56984b22-0667-4b89-aded-255238682def-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56984b22-0667-4b89-aded-255238682def-maas-api-xks-ca\") pod \"56984b22-0667-4b89-aded-255238682def\" (UID: \"56984b22-0667-4b89-aded-255238682def\") "
Nov 02 10:21:11.496826 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:11.496794    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56984b22-0667-4b89-aded-255238682def-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56984b22-0667-4b89-aded-255238682def-maas-api-openshift-ca\") pod \"56984b22-0667-4b89-aded-255238682def\" (UID: \"56984b22-0667-4b89-aded-255238682def\") "
Nov 02 10:21:11.497154 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:11.497131    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56984b22-0667-4b89-aded-255238682def-maas-api-openshift-ca" pod "56984b22-0667-4b89-aded-255238682def" (UID: "56984b22-0667-4b89-aded-255238682def"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:21:11.498812 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:11.498791    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56984b22-0667-4b89-aded-255238682def-kube-api-access-9t77d" pod "56984b22-0667-4b89-aded-255238682def" (UID: "56984b22-0667-4b89-aded-255238682def"). InnerVolumeSpecName "kube-api-access-9t77d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:21:11.498922 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:11.498873    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56984b22-0667-4b89-aded-255238682def-maas-api-xks-ca" pod "56984b22-0667-4b89-aded-255238682def" (UID: "56984b22-0667-4b89-aded-255238682def"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:21:11.598779 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:11.598687    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56984b22-0667-4b89-aded-255238682def-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:21:11.598779 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:11.598727    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t77d\" (UniqueName: \"kubernetes.io/projected/56984b22-0667-4b89-aded-255238682def-kube-api-access-9t77d\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:21:11.598779 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:11.598742    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56984b22-0667-4b89-aded-255238682def-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:21:12.306676 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:12.306645    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"
Nov 02 10:21:12.306676 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:12.306669    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t" event={"ID":"56984b22-0667-4b89-aded-255238682def","Type":"ContainerDied","Data":"1942ec4d9efdbba1540416b95cf2f66e6f36e19504d9c2e559189e475c1e5db6"}
Nov 02 10:21:12.306900 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:12.306713    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1942ec4d9efdbba1540416b95cf2f66e6f36e19504d9c2e559189e475c1e5db6"
Nov 02 10:21:21.817242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:21.816409    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:21:21.817242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:21.816518    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:21:23.595518 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.595472    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"]
Nov 02 10:21:23.596154 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.595982    4449 cpu_manager.go:414] "containerMap: removing container" podUID="56984b22-0667-4b89-aded-255238682def" containerName="revoke-keys"
Nov 02 10:21:23.596154 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.595996    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56984b22-0667-4b89-aded-255238682def" containerName="revoke-keys"
Nov 02 10:21:23.596154 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.596063    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="56984b22-0667-4b89-aded-255238682def" containerName="revoke-keys"
Nov 02 10:21:23.599415 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.599393    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.603928 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.603902    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-8d45bb-openshift-default-dockercfg-vkgg2\""
Nov 02 10:21:23.613383 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.613353    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"]
Nov 02 10:21:23.648811 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.648771    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-data\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.648991 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.648829    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-podinfo\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.648991 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.648904    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkq7j\" (UniqueName: \"kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-kube-api-access-pkq7j\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.648991 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.648938    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a6523a-4a18-412d-8b42-fe1e70daa302-istiod-ca-cert\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.649104 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.649012    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-envoy\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.649104 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.649078    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-certs\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.649177 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.649104    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-credential-socket\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.649243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.649191    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-token\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.649288 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.649242    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-socket\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.750712 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.750670    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-credential-socket\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.750939 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.750776    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-token\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.750939 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.750805    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-socket\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.750939 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.750840    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-data\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.750939 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.750884    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-podinfo\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.750939 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.750916    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkq7j\" (UniqueName: \"kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-kube-api-access-pkq7j\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.751260 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.750953    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a6523a-4a18-412d-8b42-fe1e70daa302-istiod-ca-cert\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.751260 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.751003    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-envoy\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.751260 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.751062    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-certs\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.751447 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.751270    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-credential-socket\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.751447 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.751373    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-data\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.751564 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.751483    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-socket\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.751564 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.751499    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-certs\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.752009 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.751986    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a6523a-4a18-412d-8b42-fe1e70daa302-istiod-ca-cert\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.754136 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.754109    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-podinfo\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.755132 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.755115    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-envoy\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.763742 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.763722    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkq7j\" (UniqueName: \"kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-kube-api-access-pkq7j\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.763742 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.763735    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-token\") pod \"e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") " pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:23.915786 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:23.915684    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:24.204905 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:24.204875    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"]
Nov 02 10:21:24.205706 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:21:24.205671    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a6523a_4a18_412d_8b42_fe1e70daa302.slice/crio-3bd3692a83a28707192923c13848d5e749249b21a5234e183f4d3897837dde59 WatchSource:0}: Error finding container 3bd3692a83a28707192923c13848d5e749249b21a5234e183f4d3897837dde59: Status 404 returned error can't find the container with id 3bd3692a83a28707192923c13848d5e749249b21a5234e183f4d3897837dde59
Nov 02 10:21:24.208011 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:24.207976    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:21:24.208087 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:24.208042    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:21:24.208087 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:24.208072    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:21:24.442840 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:24.442795    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6" event={"ID":"c8a6523a-4a18-412d-8b42-fe1e70daa302","Type":"ContainerStarted","Data":"2cdf3d0f5f0836f3dbd4560ecd35863e30a6f127679280668d4efe1e1137a75a"}
Nov 02 10:21:24.443040 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:24.442859    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6" event={"ID":"c8a6523a-4a18-412d-8b42-fe1e70daa302","Type":"ContainerStarted","Data":"3bd3692a83a28707192923c13848d5e749249b21a5234e183f4d3897837dde59"}
Nov 02 10:21:24.473749 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:24.473631    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6" podStartSLOduration=1.473610197 podStartE2EDuration="1.473610197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:21:24.471608889 +0000 UTC m=+2761.409895666" watchObservedRunningTime="2026-11-02 10:21:24.473610197 +0000 UTC m=+2761.411896979"
Nov 02 10:21:24.915946 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:24.915912    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:25.921165 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:25.921132    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:26.455337 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:26.455302    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:26.456470 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:26.456441    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:26.580099 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:26.580072    4449 scope.go:122] "RemoveContainer" containerID="7bbce12a4fa1e83e43872d0e1f1f791fb68e1a07393b512a8732cd4b45932a40"
Nov 02 10:21:36.547398 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:21:36.547357    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:21:40.563021 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:40.562990    4449 generic.go:356] "Generic (PLEG): container finished" podID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerID="2cdf3d0f5f0836f3dbd4560ecd35863e30a6f127679280668d4efe1e1137a75a" exitCode=0
Nov 02 10:21:40.563632 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:40.563069    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6" event={"ID":"c8a6523a-4a18-412d-8b42-fe1e70daa302","Type":"ContainerDied","Data":"2cdf3d0f5f0836f3dbd4560ecd35863e30a6f127679280668d4efe1e1137a75a"}
Nov 02 10:21:40.563632 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:40.563596    4449 scope.go:122] "RemoveContainer" containerID="2cdf3d0f5f0836f3dbd4560ecd35863e30a6f127679280668d4efe1e1137a75a"
Nov 02 10:21:40.564865 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:40.564837    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:21:40.564932 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:40.564906    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:21:40.564972 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:40.564940    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:21:41.463244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:41.460067    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:41.580172 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:41.580119    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6" event={"ID":"c8a6523a-4a18-412d-8b42-fe1e70daa302","Type":"ContainerStarted","Data":"deba8881c318e0a3a20a824b5c9699f5acf5b28bcb7c3f5fd9e0442bc1c3f45c"}
Nov 02 10:21:41.916856 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:41.916819    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:41.917934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:41.917909    4449 patch_prober.go:29] interesting pod/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body=
Nov 02 10:21:41.918019 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:41.917962    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6" podUID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused"
Nov 02 10:21:42.916359 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:42.916314    4449 patch_prober.go:29] interesting pod/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body=
Nov 02 10:21:42.916910 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:42.916383    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6" podUID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused"
Nov 02 10:21:43.807385 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:43.807346    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"]
Nov 02 10:21:43.811343 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:43.811314    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:43.817546 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:43.817518    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8d45bb\"/\"openshift-service-ca.crt\""
Nov 02 10:21:43.818835 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:43.818789    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8d45bb\"/\"kube-root-ca.crt\""
Nov 02 10:21:43.819117 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:43.819099    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8d45bb\"/\"default-dockercfg-kzm47\""
Nov 02 10:21:43.819322 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:43.819300    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8d45bb\"/\"simulator-single-main-8d45bb-kserve-self-signed-certs\""
Nov 02 10:21:43.824266 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:43.824240    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"]
Nov 02 10:21:43.916177 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:43.916141    4449 patch_prober.go:29] interesting pod/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body=
Nov 02 10:21:43.916368 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:43.916212    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6" podUID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused"
Nov 02 10:21:44.002086 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.002004    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-model-cache\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.002603 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.002200    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-tmp-dir\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.002603 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.002342    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmldl\" (UniqueName: \"kubernetes.io/projected/97fb7595-d6bf-4556-9ff8-d911172a103e-kube-api-access-dmldl\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.002603 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.002392    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-dshm\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.002603 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.002472    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-home\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.002603 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.002587    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fb7595-d6bf-4556-9ff8-d911172a103e-tls-certs\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.104313 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.104168    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fb7595-d6bf-4556-9ff8-d911172a103e-tls-certs\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.104313 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.104297    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-model-cache\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.104583 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.104372    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-tmp-dir\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.104583 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.104418    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmldl\" (UniqueName: \"kubernetes.io/projected/97fb7595-d6bf-4556-9ff8-d911172a103e-kube-api-access-dmldl\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.104583 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.104448    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-dshm\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.104583 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.104480    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-home\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.104813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.104764    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-model-cache\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.104813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.104780    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-tmp-dir\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.104935 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.104886    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-home\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.106812 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.106785    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-dshm\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.107052 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.107032    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fb7595-d6bf-4556-9ff8-d911172a103e-tls-certs\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.117743 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.117717    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmldl\" (UniqueName: \"kubernetes.io/projected/97fb7595-d6bf-4556-9ff8-d911172a103e-kube-api-access-dmldl\") pod \"simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") " pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.126988 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.126961    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:44.438156 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.438128    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"]
Nov 02 10:21:44.438617 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:21:44.438593    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97fb7595_d6bf_4556_9ff8_d911172a103e.slice/crio-119edc2609f6eba5c5dd8dd77c8dcd0c09e317716a914b5bb727744aaef6fa27 WatchSource:0}: Error finding container 119edc2609f6eba5c5dd8dd77c8dcd0c09e317716a914b5bb727744aaef6fa27: Status 404 returned error can't find the container with id 119edc2609f6eba5c5dd8dd77c8dcd0c09e317716a914b5bb727744aaef6fa27
Nov 02 10:21:44.601563 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.601512    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk" event={"ID":"97fb7595-d6bf-4556-9ff8-d911172a103e","Type":"ContainerStarted","Data":"06834e5fd1eaa9058863bd47a904df0576f9b1bf1acf8ed4155a3fbba8ded31a"}
Nov 02 10:21:44.601563 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.601563    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk" event={"ID":"97fb7595-d6bf-4556-9ff8-d911172a103e","Type":"ContainerStarted","Data":"119edc2609f6eba5c5dd8dd77c8dcd0c09e317716a914b5bb727744aaef6fa27"}
Nov 02 10:21:44.621887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.621827    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk" podStartSLOduration=1.621813578 podStartE2EDuration="1.621813578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:21:44.620147528 +0000 UTC m=+2781.558434305" watchObservedRunningTime="2026-11-02 10:21:44.621813578 +0000 UTC m=+2781.560100404"
Nov 02 10:21:44.916846 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.916810    4449 patch_prober.go:29] interesting pod/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body=
Nov 02 10:21:44.917031 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:44.916866    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6" podUID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused"
Nov 02 10:21:45.127570 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:45.127534    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:45.916690 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:45.916653    4449 patch_prober.go:29] interesting pod/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body=
Nov 02 10:21:45.916887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:45.916711    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6" podUID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused"
Nov 02 10:21:47.255721 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:47.255656    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:47.256574 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:47.256512    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:47.257272 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:47.257199    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:21:54.127117 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:54.127080    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:54.141807 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:54.141775    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:21:54.685761 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:21:54.685732    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:22:06.546770 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:22:06.546729    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:22:13.929716 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:13.929685    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"]
Nov 02 10:22:13.933916 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:13.933892    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"
Nov 02 10:22:13.942128 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:13.941643    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:22:13.942128 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:13.941736    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:22:13.942319 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:13.942307    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:22:13.942461 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:13.942436    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:22:13.944894 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:13.944863    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"]
Nov 02 10:22:14.027355 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:14.027319    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47xdb\" (UniqueName: \"kubernetes.io/projected/08d3743d-cb44-40bd-9844-19a2b82eec63-kube-api-access-47xdb\") pod \"maas-api-revoke-sub-a1bf7e6b-7rgfb\" (UID: \"08d3743d-cb44-40bd-9844-19a2b82eec63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"
Nov 02 10:22:14.027541 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:14.027455    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1bf7e6b-7rgfb\" (UID: \"08d3743d-cb44-40bd-9844-19a2b82eec63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"
Nov 02 10:22:14.027541 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:14.027501    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1bf7e6b-7rgfb\" (UID: \"08d3743d-cb44-40bd-9844-19a2b82eec63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"
Nov 02 10:22:14.128345 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:14.128307    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1bf7e6b-7rgfb\" (UID: \"08d3743d-cb44-40bd-9844-19a2b82eec63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"
Nov 02 10:22:14.128565 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:14.128458    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47xdb\" (UniqueName: \"kubernetes.io/projected/08d3743d-cb44-40bd-9844-19a2b82eec63-kube-api-access-47xdb\") pod \"maas-api-revoke-sub-a1bf7e6b-7rgfb\" (UID: \"08d3743d-cb44-40bd-9844-19a2b82eec63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"
Nov 02 10:22:14.128565 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:14.128560    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1bf7e6b-7rgfb\" (UID: \"08d3743d-cb44-40bd-9844-19a2b82eec63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"
Nov 02 10:22:14.129377 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:14.129353    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1bf7e6b-7rgfb\" (UID: \"08d3743d-cb44-40bd-9844-19a2b82eec63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"
Nov 02 10:22:14.130821 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:14.130800    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1bf7e6b-7rgfb\" (UID: \"08d3743d-cb44-40bd-9844-19a2b82eec63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"
Nov 02 10:22:14.141307 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:14.141279    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47xdb\" (UniqueName: \"kubernetes.io/projected/08d3743d-cb44-40bd-9844-19a2b82eec63-kube-api-access-47xdb\") pod \"maas-api-revoke-sub-a1bf7e6b-7rgfb\" (UID: \"08d3743d-cb44-40bd-9844-19a2b82eec63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"
Nov 02 10:22:14.250381 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:14.250289    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"
Nov 02 10:22:14.529119 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:14.529090    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"]
Nov 02 10:22:14.530424 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:22:14.530398    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d3743d_cb44_40bd_9844_19a2b82eec63.slice/crio-e87b04b72a63f7cbd91711299f0bf2a6d93554c5c819bd963dd202787ba22911 WatchSource:0}: Error finding container e87b04b72a63f7cbd91711299f0bf2a6d93554c5c819bd963dd202787ba22911: Status 404 returned error can't find the container with id e87b04b72a63f7cbd91711299f0bf2a6d93554c5c819bd963dd202787ba22911
Nov 02 10:22:14.807663 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:14.807616    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb" event={"ID":"08d3743d-cb44-40bd-9844-19a2b82eec63","Type":"ContainerStarted","Data":"e87b04b72a63f7cbd91711299f0bf2a6d93554c5c819bd963dd202787ba22911"}
Nov 02 10:22:15.816588 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:15.816556    4449 generic.go:356] "Generic (PLEG): container finished" podID="08d3743d-cb44-40bd-9844-19a2b82eec63" containerID="c91ef0659e5a4e6c52038ffcc8f15d102e866190c5c22cec94482412d0bf6ac3" exitCode=0
Nov 02 10:22:15.816999 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:15.816613    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb" event={"ID":"08d3743d-cb44-40bd-9844-19a2b82eec63","Type":"ContainerDied","Data":"c91ef0659e5a4e6c52038ffcc8f15d102e866190c5c22cec94482412d0bf6ac3"}
Nov 02 10:22:16.961333 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:16.961306    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"
Nov 02 10:22:17.059246 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:17.059169    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-xks-ca\") pod \"08d3743d-cb44-40bd-9844-19a2b82eec63\" (UID: \"08d3743d-cb44-40bd-9844-19a2b82eec63\") "
Nov 02 10:22:17.059457 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:17.059302    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d3743d-cb44-40bd-9844-19a2b82eec63-kube-api-access-47xdb\" (UniqueName: \"kubernetes.io/projected/08d3743d-cb44-40bd-9844-19a2b82eec63-kube-api-access-47xdb\") pod \"08d3743d-cb44-40bd-9844-19a2b82eec63\" (UID: \"08d3743d-cb44-40bd-9844-19a2b82eec63\") "
Nov 02 10:22:17.059457 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:17.059427    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-openshift-ca\") pod \"08d3743d-cb44-40bd-9844-19a2b82eec63\" (UID: \"08d3743d-cb44-40bd-9844-19a2b82eec63\") "
Nov 02 10:22:17.060117 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:17.060069    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-openshift-ca" pod "08d3743d-cb44-40bd-9844-19a2b82eec63" (UID: "08d3743d-cb44-40bd-9844-19a2b82eec63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:22:17.062139 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:17.062112    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-xks-ca" pod "08d3743d-cb44-40bd-9844-19a2b82eec63" (UID: "08d3743d-cb44-40bd-9844-19a2b82eec63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:22:17.062269 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:17.062152    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d3743d-cb44-40bd-9844-19a2b82eec63-kube-api-access-47xdb" pod "08d3743d-cb44-40bd-9844-19a2b82eec63" (UID: "08d3743d-cb44-40bd-9844-19a2b82eec63"). InnerVolumeSpecName "kube-api-access-47xdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:22:17.161240 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:17.161134    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:22:17.161240 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:17.161166    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47xdb\" (UniqueName: \"kubernetes.io/projected/08d3743d-cb44-40bd-9844-19a2b82eec63-kube-api-access-47xdb\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:22:17.161240 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:17.161177    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d3743d-cb44-40bd-9844-19a2b82eec63-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:22:17.830770 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:17.830744    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"
Nov 02 10:22:17.830977 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:17.830781    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb" event={"ID":"08d3743d-cb44-40bd-9844-19a2b82eec63","Type":"ContainerDied","Data":"e87b04b72a63f7cbd91711299f0bf2a6d93554c5c819bd963dd202787ba22911"}
Nov 02 10:22:17.830977 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:17.830837    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e87b04b72a63f7cbd91711299f0bf2a6d93554c5c819bd963dd202787ba22911"
Nov 02 10:22:25.031800 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:25.031756    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"]
Nov 02 10:22:25.032431 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:25.031822    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-nf7kf"]
Nov 02 10:22:25.206785 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:25.206740    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c45a9c5-2592-4fb7-9807-bfa772dd83fa" path="/var/lib/kubelet/pods/4c45a9c5-2592-4fb7-9807-bfa772dd83fa/volumes"
Nov 02 10:22:29.744455 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:29.744414    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"]
Nov 02 10:22:36.546604 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:22:36.546523    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:22:42.026478 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:42.026434    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"]
Nov 02 10:22:42.030591 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:42.030561    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hnhvm"]
Nov 02 10:22:43.205662 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:43.205615    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca92f266-e97e-4009-9cc6-5d138a1c9a92" path="/var/lib/kubelet/pods/ca92f266-e97e-4009-9cc6-5d138a1c9a92/volumes"
Nov 02 10:22:44.771659 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:44.771614    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk" podUID="97fb7595-d6bf-4556-9ff8-d911172a103e" containerName="main" containerID="cri-o://06834e5fd1eaa9058863bd47a904df0576f9b1bf1acf8ed4155a3fbba8ded31a" gracePeriod=45
Nov 02 10:22:45.010093 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.010061    4449 generic.go:356] "Generic (PLEG): container finished" podID="97fb7595-d6bf-4556-9ff8-d911172a103e" containerID="06834e5fd1eaa9058863bd47a904df0576f9b1bf1acf8ed4155a3fbba8ded31a" exitCode=0
Nov 02 10:22:45.010272 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.010133    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk" event={"ID":"97fb7595-d6bf-4556-9ff8-d911172a103e","Type":"ContainerDied","Data":"06834e5fd1eaa9058863bd47a904df0576f9b1bf1acf8ed4155a3fbba8ded31a"}
Nov 02 10:22:45.031101 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.031038    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:22:45.134438 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.134404    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-tmp-dir\") pod \"97fb7595-d6bf-4556-9ff8-d911172a103e\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") "
Nov 02 10:22:45.134617 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.134452    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fb7595-d6bf-4556-9ff8-d911172a103e-kube-api-access-dmldl\" (UniqueName: \"kubernetes.io/projected/97fb7595-d6bf-4556-9ff8-d911172a103e-kube-api-access-dmldl\") pod \"97fb7595-d6bf-4556-9ff8-d911172a103e\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") "
Nov 02 10:22:45.134617 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.134481    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-home\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-home\") pod \"97fb7595-d6bf-4556-9ff8-d911172a103e\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") "
Nov 02 10:22:45.134617 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.134579    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-model-cache\") pod \"97fb7595-d6bf-4556-9ff8-d911172a103e\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") "
Nov 02 10:22:45.134617 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.134602    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97fb7595-d6bf-4556-9ff8-d911172a103e-tls-certs\" (UniqueName: \"kubernetes.io/secret/97fb7595-d6bf-4556-9ff8-d911172a103e-tls-certs\") pod \"97fb7595-d6bf-4556-9ff8-d911172a103e\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") "
Nov 02 10:22:45.134810 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.134683    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-dshm\") pod \"97fb7595-d6bf-4556-9ff8-d911172a103e\" (UID: \"97fb7595-d6bf-4556-9ff8-d911172a103e\") "
Nov 02 10:22:45.134810 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.134724    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-tmp-dir" pod "97fb7595-d6bf-4556-9ff8-d911172a103e" (UID: "97fb7595-d6bf-4556-9ff8-d911172a103e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:22:45.134929 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.134858    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-model-cache" pod "97fb7595-d6bf-4556-9ff8-d911172a103e" (UID: "97fb7595-d6bf-4556-9ff8-d911172a103e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:22:45.134929 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.134870    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-home" pod "97fb7595-d6bf-4556-9ff8-d911172a103e" (UID: "97fb7595-d6bf-4556-9ff8-d911172a103e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:22:45.135051 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.134991    4449 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-tmp-dir\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:22:45.135051 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.135004    4449 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-home\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:22:45.135051 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.135012    4449 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-model-cache\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:22:45.137033 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.137005    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-dshm" pod "97fb7595-d6bf-4556-9ff8-d911172a103e" (UID: "97fb7595-d6bf-4556-9ff8-d911172a103e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:22:45.137333 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.137309    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97fb7595-d6bf-4556-9ff8-d911172a103e-tls-certs" pod "97fb7595-d6bf-4556-9ff8-d911172a103e" (UID: "97fb7595-d6bf-4556-9ff8-d911172a103e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:22:45.137439 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.137361    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fb7595-d6bf-4556-9ff8-d911172a103e-kube-api-access-dmldl" pod "97fb7595-d6bf-4556-9ff8-d911172a103e" (UID: "97fb7595-d6bf-4556-9ff8-d911172a103e"). InnerVolumeSpecName "kube-api-access-dmldl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:22:45.235861 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.235833    4449 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fb7595-d6bf-4556-9ff8-d911172a103e-tls-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:22:45.235861 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.235858    4449 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fb7595-d6bf-4556-9ff8-d911172a103e-dshm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:22:45.236033 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:45.235868    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmldl\" (UniqueName: \"kubernetes.io/projected/97fb7595-d6bf-4556-9ff8-d911172a103e-kube-api-access-dmldl\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:22:46.017726 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:46.017691    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"
Nov 02 10:22:46.018150 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:46.017691    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk" event={"ID":"97fb7595-d6bf-4556-9ff8-d911172a103e","Type":"ContainerDied","Data":"119edc2609f6eba5c5dd8dd77c8dcd0c09e317716a914b5bb727744aaef6fa27"}
Nov 02 10:22:46.018150 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:46.017830    4449 scope.go:122] "RemoveContainer" containerID="06834e5fd1eaa9058863bd47a904df0576f9b1bf1acf8ed4155a3fbba8ded31a"
Nov 02 10:22:46.041343 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:46.041314    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"]
Nov 02 10:22:46.045460 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:46.045425    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-8d45bb/simulator-single-main-8d45bb-kserve-7f5679d7c7-k25rk"]
Nov 02 10:22:47.206028 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:47.205992    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97fb7595-d6bf-4556-9ff8-d911172a103e" path="/var/lib/kubelet/pods/97fb7595-d6bf-4556-9ff8-d911172a103e/volumes"
Nov 02 10:22:59.026372 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:59.026328    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"]
Nov 02 10:22:59.031381 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:59.031351    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6hthp"]
Nov 02 10:22:59.206083 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:22:59.206051    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2" path="/var/lib/kubelet/pods/9b561a93-0f0a-40f7-adf4-b3bfcf60ccb2/volumes"
Nov 02 10:23:06.547982 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:23:06.547944    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:23:10.024968 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:10.024931    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"]
Nov 02 10:23:10.028068 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:10.028043    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qbg69"]
Nov 02 10:23:11.205807 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:11.205766    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fadfacd0-e678-42c4-8188-692802062b74" path="/var/lib/kubelet/pods/fadfacd0-e678-42c4-8188-692802062b74/volumes"
Nov 02 10:23:13.189752 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.189710    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"]
Nov 02 10:23:13.190428 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.190407    4449 cpu_manager.go:414] "containerMap: removing container" podUID="08d3743d-cb44-40bd-9844-19a2b82eec63" containerName="revoke-keys"
Nov 02 10:23:13.190535 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.190429    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d3743d-cb44-40bd-9844-19a2b82eec63" containerName="revoke-keys"
Nov 02 10:23:13.190535 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.190475    4449 cpu_manager.go:414] "containerMap: removing container" podUID="97fb7595-d6bf-4556-9ff8-d911172a103e" containerName="main"
Nov 02 10:23:13.190535 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.190484    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fb7595-d6bf-4556-9ff8-d911172a103e" containerName="main"
Nov 02 10:23:13.190707 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.190586    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d3743d-cb44-40bd-9844-19a2b82eec63" containerName="revoke-keys"
Nov 02 10:23:13.190707 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.190606    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fb7595-d6bf-4556-9ff8-d911172a103e" containerName="main"
Nov 02 10:23:13.194230 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.194189    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"
Nov 02 10:23:13.200261 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.200234    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:23:13.200261 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.200247    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:23:13.201329 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.200769    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:23:13.201329 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.201301    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:23:13.208960 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.208933    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"]
Nov 02 10:23:13.314962 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.314925    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg\" (UID: \"4b629496-4fe2-4c86-b990-357b92e56211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"
Nov 02 10:23:13.315189 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.314984    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg\" (UID: \"4b629496-4fe2-4c86-b990-357b92e56211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"
Nov 02 10:23:13.315189 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.315173    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzfs6\" (UniqueName: \"kubernetes.io/projected/4b629496-4fe2-4c86-b990-357b92e56211-kube-api-access-wzfs6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg\" (UID: \"4b629496-4fe2-4c86-b990-357b92e56211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"
Nov 02 10:23:13.416310 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.416268    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg\" (UID: \"4b629496-4fe2-4c86-b990-357b92e56211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"
Nov 02 10:23:13.416500 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.416388    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzfs6\" (UniqueName: \"kubernetes.io/projected/4b629496-4fe2-4c86-b990-357b92e56211-kube-api-access-wzfs6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg\" (UID: \"4b629496-4fe2-4c86-b990-357b92e56211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"
Nov 02 10:23:13.416500 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.416490    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg\" (UID: \"4b629496-4fe2-4c86-b990-357b92e56211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"
Nov 02 10:23:13.416993 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.416973    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg\" (UID: \"4b629496-4fe2-4c86-b990-357b92e56211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"
Nov 02 10:23:13.419294 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.419266    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg\" (UID: \"4b629496-4fe2-4c86-b990-357b92e56211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"
Nov 02 10:23:13.430065 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.430036    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzfs6\" (UniqueName: \"kubernetes.io/projected/4b629496-4fe2-4c86-b990-357b92e56211-kube-api-access-wzfs6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg\" (UID: \"4b629496-4fe2-4c86-b990-357b92e56211\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"
Nov 02 10:23:13.510323 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.510233    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"
Nov 02 10:23:13.787543 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:13.787504    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"]
Nov 02 10:23:13.788569 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:23:13.788543    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b629496_4fe2_4c86_b990_357b92e56211.slice/crio-c885eb4b3fe0bfb0081c6cec36a208b2fcf2662f0d6a131d34b67e574aafe742 WatchSource:0}: Error finding container c885eb4b3fe0bfb0081c6cec36a208b2fcf2662f0d6a131d34b67e574aafe742: Status 404 returned error can't find the container with id c885eb4b3fe0bfb0081c6cec36a208b2fcf2662f0d6a131d34b67e574aafe742
Nov 02 10:23:14.198937 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:14.198887    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg" event={"ID":"4b629496-4fe2-4c86-b990-357b92e56211","Type":"ContainerStarted","Data":"fd9207633574caea0ddd46be861eee7245d75fc69f6ed59e151e36b0ecdf9f70"}
Nov 02 10:23:14.199536 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:14.199006    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg" event={"ID":"4b629496-4fe2-4c86-b990-357b92e56211","Type":"ContainerStarted","Data":"c885eb4b3fe0bfb0081c6cec36a208b2fcf2662f0d6a131d34b67e574aafe742"}
Nov 02 10:23:14.220147 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:14.220088    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg" podStartSLOduration=1.2200751730000001 podStartE2EDuration="1.220075173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:23:14.217846125 +0000 UTC m=+2871.156132918" watchObservedRunningTime="2026-11-02 10:23:14.220075173 +0000 UTC m=+2871.158361949"
Nov 02 10:23:15.208109 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:15.208073    4449 generic.go:356] "Generic (PLEG): container finished" podID="4b629496-4fe2-4c86-b990-357b92e56211" containerID="fd9207633574caea0ddd46be861eee7245d75fc69f6ed59e151e36b0ecdf9f70" exitCode=0
Nov 02 10:23:15.208621 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:15.208145    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg" event={"ID":"4b629496-4fe2-4c86-b990-357b92e56211","Type":"ContainerDied","Data":"fd9207633574caea0ddd46be861eee7245d75fc69f6ed59e151e36b0ecdf9f70"}
Nov 02 10:23:16.359600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:16.359565    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"
Nov 02 10:23:16.449204 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:16.449127    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b629496-4fe2-4c86-b990-357b92e56211-kube-api-access-wzfs6\" (UniqueName: \"kubernetes.io/projected/4b629496-4fe2-4c86-b990-357b92e56211-kube-api-access-wzfs6\") pod \"4b629496-4fe2-4c86-b990-357b92e56211\" (UID: \"4b629496-4fe2-4c86-b990-357b92e56211\") "
Nov 02 10:23:16.449204 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:16.449191    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-xks-ca\") pod \"4b629496-4fe2-4c86-b990-357b92e56211\" (UID: \"4b629496-4fe2-4c86-b990-357b92e56211\") "
Nov 02 10:23:16.449416 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:16.449329    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-openshift-ca\") pod \"4b629496-4fe2-4c86-b990-357b92e56211\" (UID: \"4b629496-4fe2-4c86-b990-357b92e56211\") "
Nov 02 10:23:16.449735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:16.449705    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-openshift-ca" pod "4b629496-4fe2-4c86-b990-357b92e56211" (UID: "4b629496-4fe2-4c86-b990-357b92e56211"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:23:16.451475 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:16.451443    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-xks-ca" pod "4b629496-4fe2-4c86-b990-357b92e56211" (UID: "4b629496-4fe2-4c86-b990-357b92e56211"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:23:16.451566 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:16.451534    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b629496-4fe2-4c86-b990-357b92e56211-kube-api-access-wzfs6" pod "4b629496-4fe2-4c86-b990-357b92e56211" (UID: "4b629496-4fe2-4c86-b990-357b92e56211"). InnerVolumeSpecName "kube-api-access-wzfs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:23:16.551305 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:16.551266    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:16.551305 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:16.551303    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzfs6\" (UniqueName: \"kubernetes.io/projected/4b629496-4fe2-4c86-b990-357b92e56211-kube-api-access-wzfs6\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:16.551305 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:16.551313    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b629496-4fe2-4c86-b990-357b92e56211-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:17.220650 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:17.220624    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"
Nov 02 10:23:17.220794 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:17.220621    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg" event={"ID":"4b629496-4fe2-4c86-b990-357b92e56211","Type":"ContainerDied","Data":"c885eb4b3fe0bfb0081c6cec36a208b2fcf2662f0d6a131d34b67e574aafe742"}
Nov 02 10:23:17.220794 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:17.220765    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c885eb4b3fe0bfb0081c6cec36a208b2fcf2662f0d6a131d34b67e574aafe742"
Nov 02 10:23:24.677054 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:24.677012    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"]
Nov 02 10:23:24.680485 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:24.680459    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8d45bb-vxbvg"]
Nov 02 10:23:25.048290 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:25.048258    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"]
Nov 02 10:23:25.048610 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:25.048557    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6" podUID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerName="istio-proxy" containerID="cri-o://deba8881c318e0a3a20a824b5c9699f5acf5b28bcb7c3f5fd9e0442bc1c3f45c" gracePeriod=30
Nov 02 10:23:25.206491 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:25.206453    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b629496-4fe2-4c86-b990-357b92e56211" path="/var/lib/kubelet/pods/4b629496-4fe2-4c86-b990-357b92e56211/volumes"
Nov 02 10:23:26.318777 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.318744    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"]
Nov 02 10:23:26.327766 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.327736    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jj42t"]
Nov 02 10:23:26.346806 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.346775    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"]
Nov 02 10:23:26.347238 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.347211    4449 cpu_manager.go:414] "containerMap: removing container" podUID="4b629496-4fe2-4c86-b990-357b92e56211" containerName="revoke-keys"
Nov 02 10:23:26.347305 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.347239    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b629496-4fe2-4c86-b990-357b92e56211" containerName="revoke-keys"
Nov 02 10:23:26.347345 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.347339    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b629496-4fe2-4c86-b990-357b92e56211" containerName="revoke-keys"
Nov 02 10:23:26.350894 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.350876    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"
Nov 02 10:23:26.357444 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.357414    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:23:26.357747 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.357723    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:23:26.357857 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.357786    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:23:26.357857 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.357846    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:23:26.361729 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.361705    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"]
Nov 02 10:23:26.449594 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.449554    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-255gc\" (UniqueName: \"kubernetes.io/projected/db040ad0-591e-409c-80cd-83d594857877-kube-api-access-255gc\") pod \"maas-api-revoke-sub-3d4156f1-6r776\" (UID: \"db040ad0-591e-409c-80cd-83d594857877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"
Nov 02 10:23:26.449754 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.449616    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db040ad0-591e-409c-80cd-83d594857877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6r776\" (UID: \"db040ad0-591e-409c-80cd-83d594857877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"
Nov 02 10:23:26.449754 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.449730    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db040ad0-591e-409c-80cd-83d594857877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6r776\" (UID: \"db040ad0-591e-409c-80cd-83d594857877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"
Nov 02 10:23:26.550686 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.550650    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db040ad0-591e-409c-80cd-83d594857877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6r776\" (UID: \"db040ad0-591e-409c-80cd-83d594857877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"
Nov 02 10:23:26.550887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.550719    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db040ad0-591e-409c-80cd-83d594857877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6r776\" (UID: \"db040ad0-591e-409c-80cd-83d594857877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"
Nov 02 10:23:26.550887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.550792    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-255gc\" (UniqueName: \"kubernetes.io/projected/db040ad0-591e-409c-80cd-83d594857877-kube-api-access-255gc\") pod \"maas-api-revoke-sub-3d4156f1-6r776\" (UID: \"db040ad0-591e-409c-80cd-83d594857877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"
Nov 02 10:23:26.551562 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.551536    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db040ad0-591e-409c-80cd-83d594857877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6r776\" (UID: \"db040ad0-591e-409c-80cd-83d594857877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"
Nov 02 10:23:26.553342 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.553314    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db040ad0-591e-409c-80cd-83d594857877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6r776\" (UID: \"db040ad0-591e-409c-80cd-83d594857877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"
Nov 02 10:23:26.563703 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.563677    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-255gc\" (UniqueName: \"kubernetes.io/projected/db040ad0-591e-409c-80cd-83d594857877-kube-api-access-255gc\") pod \"maas-api-revoke-sub-3d4156f1-6r776\" (UID: \"db040ad0-591e-409c-80cd-83d594857877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"
Nov 02 10:23:26.618088 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.618004    4449 scope.go:122] "RemoveContainer" containerID="b1bcb40e2530f80719c8dc65e85d9843c168be03514e4b948102effc12b1ccf8"
Nov 02 10:23:26.667354 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.667321    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"
Nov 02 10:23:26.953626 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:26.953594    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"]
Nov 02 10:23:26.954882 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:23:26.954856    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb040ad0_591e_409c_80cd_83d594857877.slice/crio-f61eeea9175e3d4a6d0865fc4eeecf9580374aba0906f718ff57b05b11ca451f WatchSource:0}: Error finding container f61eeea9175e3d4a6d0865fc4eeecf9580374aba0906f718ff57b05b11ca451f: Status 404 returned error can't find the container with id f61eeea9175e3d4a6d0865fc4eeecf9580374aba0906f718ff57b05b11ca451f
Nov 02 10:23:27.206060 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:27.205969    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56984b22-0667-4b89-aded-255238682def" path="/var/lib/kubelet/pods/56984b22-0667-4b89-aded-255238682def/volumes"
Nov 02 10:23:27.287396 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:27.287355    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776" event={"ID":"db040ad0-591e-409c-80cd-83d594857877","Type":"ContainerStarted","Data":"eb92d0cf086a3f0d55ec6fccc79ff3259975c0153fba61643b61bf3f8ac0177a"}
Nov 02 10:23:27.287544 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:27.287456    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776" event={"ID":"db040ad0-591e-409c-80cd-83d594857877","Type":"ContainerStarted","Data":"f61eeea9175e3d4a6d0865fc4eeecf9580374aba0906f718ff57b05b11ca451f"}
Nov 02 10:23:27.309939 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:27.309889    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776" podStartSLOduration=1.309873975 podStartE2EDuration="1.309873975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:23:27.307086776 +0000 UTC m=+2884.245373551" watchObservedRunningTime="2026-11-02 10:23:27.309873975 +0000 UTC m=+2884.248160750"
Nov 02 10:23:28.295275 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:28.295236    4449 generic.go:356] "Generic (PLEG): container finished" podID="db040ad0-591e-409c-80cd-83d594857877" containerID="eb92d0cf086a3f0d55ec6fccc79ff3259975c0153fba61643b61bf3f8ac0177a" exitCode=0
Nov 02 10:23:28.295688 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:28.295309    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776" event={"ID":"db040ad0-591e-409c-80cd-83d594857877","Type":"ContainerDied","Data":"eb92d0cf086a3f0d55ec6fccc79ff3259975c0153fba61643b61bf3f8ac0177a"}
Nov 02 10:23:29.445912 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:29.445883    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"
Nov 02 10:23:29.584331 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:29.584237    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db040ad0-591e-409c-80cd-83d594857877-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db040ad0-591e-409c-80cd-83d594857877-maas-api-openshift-ca\") pod \"db040ad0-591e-409c-80cd-83d594857877\" (UID: \"db040ad0-591e-409c-80cd-83d594857877\") "
Nov 02 10:23:29.584510 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:29.584374    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db040ad0-591e-409c-80cd-83d594857877-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db040ad0-591e-409c-80cd-83d594857877-maas-api-xks-ca\") pod \"db040ad0-591e-409c-80cd-83d594857877\" (UID: \"db040ad0-591e-409c-80cd-83d594857877\") "
Nov 02 10:23:29.584589 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:29.584531    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db040ad0-591e-409c-80cd-83d594857877-kube-api-access-255gc\" (UniqueName: \"kubernetes.io/projected/db040ad0-591e-409c-80cd-83d594857877-kube-api-access-255gc\") pod \"db040ad0-591e-409c-80cd-83d594857877\" (UID: \"db040ad0-591e-409c-80cd-83d594857877\") "
Nov 02 10:23:29.584653 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:29.584622    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db040ad0-591e-409c-80cd-83d594857877-maas-api-openshift-ca" pod "db040ad0-591e-409c-80cd-83d594857877" (UID: "db040ad0-591e-409c-80cd-83d594857877"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:23:29.584871 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:29.584854    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db040ad0-591e-409c-80cd-83d594857877-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:29.586617 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:29.586587    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db040ad0-591e-409c-80cd-83d594857877-maas-api-xks-ca" pod "db040ad0-591e-409c-80cd-83d594857877" (UID: "db040ad0-591e-409c-80cd-83d594857877"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:23:29.586742 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:29.586646    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db040ad0-591e-409c-80cd-83d594857877-kube-api-access-255gc" pod "db040ad0-591e-409c-80cd-83d594857877" (UID: "db040ad0-591e-409c-80cd-83d594857877"). InnerVolumeSpecName "kube-api-access-255gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:23:29.685971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:29.685931    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db040ad0-591e-409c-80cd-83d594857877-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:29.685971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:29.685963    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-255gc\" (UniqueName: \"kubernetes.io/projected/db040ad0-591e-409c-80cd-83d594857877-kube-api-access-255gc\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:30.300228 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.300186    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:23:30.310168 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.310135    4449 generic.go:356] "Generic (PLEG): container finished" podID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerID="deba8881c318e0a3a20a824b5c9699f5acf5b28bcb7c3f5fd9e0442bc1c3f45c" exitCode=0
Nov 02 10:23:30.310370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.310186    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"
Nov 02 10:23:30.310370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.310234    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6" event={"ID":"c8a6523a-4a18-412d-8b42-fe1e70daa302","Type":"ContainerDied","Data":"deba8881c318e0a3a20a824b5c9699f5acf5b28bcb7c3f5fd9e0442bc1c3f45c"}
Nov 02 10:23:30.310370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.310288    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6" event={"ID":"c8a6523a-4a18-412d-8b42-fe1e70daa302","Type":"ContainerDied","Data":"3bd3692a83a28707192923c13848d5e749249b21a5234e183f4d3897837dde59"}
Nov 02 10:23:30.310370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.310312    4449 scope.go:122] "RemoveContainer" containerID="deba8881c318e0a3a20a824b5c9699f5acf5b28bcb7c3f5fd9e0442bc1c3f45c"
Nov 02 10:23:30.311908 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.311891    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"
Nov 02 10:23:30.312001 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.311919    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776" event={"ID":"db040ad0-591e-409c-80cd-83d594857877","Type":"ContainerDied","Data":"f61eeea9175e3d4a6d0865fc4eeecf9580374aba0906f718ff57b05b11ca451f"}
Nov 02 10:23:30.312001 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.311960    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f61eeea9175e3d4a6d0865fc4eeecf9580374aba0906f718ff57b05b11ca451f"
Nov 02 10:23:30.321487 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.321464    4449 scope.go:122] "RemoveContainer" containerID="2cdf3d0f5f0836f3dbd4560ecd35863e30a6f127679280668d4efe1e1137a75a"
Nov 02 10:23:30.333481 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.333462    4449 scope.go:122] "RemoveContainer" containerID="deba8881c318e0a3a20a824b5c9699f5acf5b28bcb7c3f5fd9e0442bc1c3f45c"
Nov 02 10:23:30.333787 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:23:30.333759    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deba8881c318e0a3a20a824b5c9699f5acf5b28bcb7c3f5fd9e0442bc1c3f45c\": container with ID starting with deba8881c318e0a3a20a824b5c9699f5acf5b28bcb7c3f5fd9e0442bc1c3f45c not found: ID does not exist" containerID="deba8881c318e0a3a20a824b5c9699f5acf5b28bcb7c3f5fd9e0442bc1c3f45c"
Nov 02 10:23:30.333908 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.333796    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deba8881c318e0a3a20a824b5c9699f5acf5b28bcb7c3f5fd9e0442bc1c3f45c"} err="failed to get container status \"deba8881c318e0a3a20a824b5c9699f5acf5b28bcb7c3f5fd9e0442bc1c3f45c\": rpc error: code = NotFound desc = could not find container \"deba8881c318e0a3a20a824b5c9699f5acf5b28bcb7c3f5fd9e0442bc1c3f45c\": container with ID starting with deba8881c318e0a3a20a824b5c9699f5acf5b28bcb7c3f5fd9e0442bc1c3f45c not found: ID does not exist"
Nov 02 10:23:30.333908 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.333813    4449 scope.go:122] "RemoveContainer" containerID="2cdf3d0f5f0836f3dbd4560ecd35863e30a6f127679280668d4efe1e1137a75a"
Nov 02 10:23:30.334114 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:23:30.334094    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cdf3d0f5f0836f3dbd4560ecd35863e30a6f127679280668d4efe1e1137a75a\": container with ID starting with 2cdf3d0f5f0836f3dbd4560ecd35863e30a6f127679280668d4efe1e1137a75a not found: ID does not exist" containerID="2cdf3d0f5f0836f3dbd4560ecd35863e30a6f127679280668d4efe1e1137a75a"
Nov 02 10:23:30.334166 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.334120    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cdf3d0f5f0836f3dbd4560ecd35863e30a6f127679280668d4efe1e1137a75a"} err="failed to get container status \"2cdf3d0f5f0836f3dbd4560ecd35863e30a6f127679280668d4efe1e1137a75a\": rpc error: code = NotFound desc = could not find container \"2cdf3d0f5f0836f3dbd4560ecd35863e30a6f127679280668d4efe1e1137a75a\": container with ID starting with 2cdf3d0f5f0836f3dbd4560ecd35863e30a6f127679280668d4efe1e1137a75a not found: ID does not exist"
Nov 02 10:23:30.392860 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.392831    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-kube-api-access-pkq7j\" (UniqueName: \"kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-kube-api-access-pkq7j\") pod \"c8a6523a-4a18-412d-8b42-fe1e70daa302\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") "
Nov 02 10:23:30.393038 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.392888    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-podinfo\") pod \"c8a6523a-4a18-412d-8b42-fe1e70daa302\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") "
Nov 02 10:23:30.393038 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.392918    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-token\" (UniqueName: \"kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-token\") pod \"c8a6523a-4a18-412d-8b42-fe1e70daa302\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") "
Nov 02 10:23:30.393038 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.392944    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-envoy\") pod \"c8a6523a-4a18-412d-8b42-fe1e70daa302\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") "
Nov 02 10:23:30.393038 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.392963    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-socket\") pod \"c8a6523a-4a18-412d-8b42-fe1e70daa302\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") "
Nov 02 10:23:30.393291 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.393037    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-credential-socket\") pod \"c8a6523a-4a18-412d-8b42-fe1e70daa302\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") "
Nov 02 10:23:30.393291 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.393063    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-data\") pod \"c8a6523a-4a18-412d-8b42-fe1e70daa302\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") "
Nov 02 10:23:30.393291 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.393100    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-certs\") pod \"c8a6523a-4a18-412d-8b42-fe1e70daa302\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") "
Nov 02 10:23:30.393291 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.393131    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a6523a-4a18-412d-8b42-fe1e70daa302-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a6523a-4a18-412d-8b42-fe1e70daa302-istiod-ca-cert\") pod \"c8a6523a-4a18-412d-8b42-fe1e70daa302\" (UID: \"c8a6523a-4a18-412d-8b42-fe1e70daa302\") "
Nov 02 10:23:30.393741 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.393709    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-data" pod "c8a6523a-4a18-412d-8b42-fe1e70daa302" (UID: "c8a6523a-4a18-412d-8b42-fe1e70daa302"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:23:30.393995 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.393793    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a6523a-4a18-412d-8b42-fe1e70daa302-istiod-ca-cert" pod "c8a6523a-4a18-412d-8b42-fe1e70daa302" (UID: "c8a6523a-4a18-412d-8b42-fe1e70daa302"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:23:30.393995 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.393886    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-credential-socket" pod "c8a6523a-4a18-412d-8b42-fe1e70daa302" (UID: "c8a6523a-4a18-412d-8b42-fe1e70daa302"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:23:30.393995 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.393956    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-certs" pod "c8a6523a-4a18-412d-8b42-fe1e70daa302" (UID: "c8a6523a-4a18-412d-8b42-fe1e70daa302"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:23:30.394190 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.394065    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-socket" pod "c8a6523a-4a18-412d-8b42-fe1e70daa302" (UID: "c8a6523a-4a18-412d-8b42-fe1e70daa302"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:23:30.395425 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.395406    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-envoy" pod "c8a6523a-4a18-412d-8b42-fe1e70daa302" (UID: "c8a6523a-4a18-412d-8b42-fe1e70daa302"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:23:30.395635 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.395611    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-kube-api-access-pkq7j" pod "c8a6523a-4a18-412d-8b42-fe1e70daa302" (UID: "c8a6523a-4a18-412d-8b42-fe1e70daa302"). InnerVolumeSpecName "kube-api-access-pkq7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:23:30.395853 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.395831    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-podinfo" pod "c8a6523a-4a18-412d-8b42-fe1e70daa302" (UID: "c8a6523a-4a18-412d-8b42-fe1e70daa302"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:23:30.395956 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.395856    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-token" pod "c8a6523a-4a18-412d-8b42-fe1e70daa302" (UID: "c8a6523a-4a18-412d-8b42-fe1e70daa302"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:23:30.494564 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.494531    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:30.494564 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.494559    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:30.494564 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.494570    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:30.495022 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.494578    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:30.495022 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.494588    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a6523a-4a18-412d-8b42-fe1e70daa302-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:30.495022 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.494596    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a6523a-4a18-412d-8b42-fe1e70daa302-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:30.495022 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.494605    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkq7j\" (UniqueName: \"kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-kube-api-access-pkq7j\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:30.495022 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.494616    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:30.495022 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.494624    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a6523a-4a18-412d-8b42-fe1e70daa302-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:23:30.635639 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.635607    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"]
Nov 02 10:23:30.640304 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:30.640273    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8d45bb-openshift-default-5b7f975b9-mz4c6"]
Nov 02 10:23:31.205962 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:31.205930    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a6523a-4a18-412d-8b42-fe1e70daa302" path="/var/lib/kubelet/pods/c8a6523a-4a18-412d-8b42-fe1e70daa302/volumes"
Nov 02 10:23:36.546526 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:23:36.546490    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:23:37.025052 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:37.024959    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"]
Nov 02 10:23:37.028136 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:37.028108    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-gn452"]
Nov 02 10:23:37.205071 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:37.205033    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5882afca-a46d-41d7-9c03-766f58607c01" path="/var/lib/kubelet/pods/5882afca-a46d-41d7-9c03-766f58607c01/volumes"
Nov 02 10:23:47.027043 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:47.026998    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"]
Nov 02 10:23:47.030366 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:47.030334    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rkrdt"]
Nov 02 10:23:47.205538 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:23:47.205494    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71952279-da18-405f-83b7-215b7cfc4540" path="/var/lib/kubelet/pods/71952279-da18-405f-83b7-215b7cfc4540/volumes"
Nov 02 10:24:06.546360 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:24:06.546276    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:24:08.034122 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:08.034056    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"]
Nov 02 10:24:08.036883 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:08.036851    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-265k6"]
Nov 02 10:24:09.206294 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:09.206248    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a07b3dc-ccdf-41cc-88ca-c9d5475cf149" path="/var/lib/kubelet/pods/6a07b3dc-ccdf-41cc-88ca-c9d5475cf149/volumes"
Nov 02 10:24:25.027436 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:25.027401    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"]
Nov 02 10:24:25.030772 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:25.030744    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qcfvv"]
Nov 02 10:24:25.205719 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:25.205686    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa1e62c5-3d95-4ebf-a9a0-1d840a002b82" path="/var/lib/kubelet/pods/aa1e62c5-3d95-4ebf-a9a0-1d840a002b82/volumes"
Nov 02 10:24:26.647905 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:26.647873    4449 scope.go:122] "RemoveContainer" containerID="501a9fa9123c80921d7ab61d66932853cc42a50e7555f63a97e2233064d4aad2"
Nov 02 10:24:26.657573 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:26.657554    4449 scope.go:122] "RemoveContainer" containerID="2c4f522eb03baec0a397b2ba3a62d88b95b3ccc281c17d16fff670ee355de376"
Nov 02 10:24:26.669365 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:26.669345    4449 scope.go:122] "RemoveContainer" containerID="c8dc3d5a91e541d3d750ddb1874b3c2c29113caac713709e5c0f6ffec0d2a2ff"
Nov 02 10:24:36.546954 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:24:36.546920    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:24:41.026412 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:41.026366    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"]
Nov 02 10:24:41.030574 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:41.030547    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-skqqb"]
Nov 02 10:24:41.205462 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:41.205422    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de73dc84-9811-4f8c-b29b-9262e818398d" path="/var/lib/kubelet/pods/de73dc84-9811-4f8c-b29b-9262e818398d/volumes"
Nov 02 10:24:55.029048 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:55.029011    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"]
Nov 02 10:24:55.031841 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:55.031811    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-h2b7t"]
Nov 02 10:24:55.208629 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:24:55.208597    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47080f32-b404-4761-a197-8c5b9e7c013c" path="/var/lib/kubelet/pods/47080f32-b404-4761-a197-8c5b9e7c013c/volumes"
Nov 02 10:25:03.645418 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:03.645361    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-24rkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:25:03.646043 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:03.645460    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:25:05.531809 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.531766    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"]
Nov 02 10:25:05.532309 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.532294    4449 cpu_manager.go:414] "containerMap: removing container" podUID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerName="istio-proxy"
Nov 02 10:25:05.532362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.532311    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerName="istio-proxy"
Nov 02 10:25:05.532362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.532337    4449 cpu_manager.go:414] "containerMap: removing container" podUID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerName="istio-proxy"
Nov 02 10:25:05.532362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.532342    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerName="istio-proxy"
Nov 02 10:25:05.532362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.532358    4449 cpu_manager.go:414] "containerMap: removing container" podUID="db040ad0-591e-409c-80cd-83d594857877" containerName="revoke-keys"
Nov 02 10:25:05.532507 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.532363    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db040ad0-591e-409c-80cd-83d594857877" containerName="revoke-keys"
Nov 02 10:25:05.532507 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.532439    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerName="istio-proxy"
Nov 02 10:25:05.532507 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.532448    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="db040ad0-591e-409c-80cd-83d594857877" containerName="revoke-keys"
Nov 02 10:25:05.536042 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.536026    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"
Nov 02 10:25:05.544175 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.544151    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:25:05.544385 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.544156    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:25:05.544502 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.544312    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:25:05.544502 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.544487    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:25:05.545486 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.545461    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"]
Nov 02 10:25:05.626358 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.626320    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-j8wtk\" (UID: \"cd345351-4fb6-41d2-b647-619bd9564d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"
Nov 02 10:25:05.626585 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.626470    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpvlt\" (UniqueName: \"kubernetes.io/projected/cd345351-4fb6-41d2-b647-619bd9564d13-kube-api-access-qpvlt\") pod \"maas-api-revoke-sub-d576e68b-j8wtk\" (UID: \"cd345351-4fb6-41d2-b647-619bd9564d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"
Nov 02 10:25:05.626585 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.626510    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-j8wtk\" (UID: \"cd345351-4fb6-41d2-b647-619bd9564d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"
Nov 02 10:25:05.727538 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.727499    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpvlt\" (UniqueName: \"kubernetes.io/projected/cd345351-4fb6-41d2-b647-619bd9564d13-kube-api-access-qpvlt\") pod \"maas-api-revoke-sub-d576e68b-j8wtk\" (UID: \"cd345351-4fb6-41d2-b647-619bd9564d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"
Nov 02 10:25:05.727751 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.727554    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-j8wtk\" (UID: \"cd345351-4fb6-41d2-b647-619bd9564d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"
Nov 02 10:25:05.727751 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.727639    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-j8wtk\" (UID: \"cd345351-4fb6-41d2-b647-619bd9564d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"
Nov 02 10:25:05.728322 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.728296    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-j8wtk\" (UID: \"cd345351-4fb6-41d2-b647-619bd9564d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"
Nov 02 10:25:05.730254 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.730230    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-j8wtk\" (UID: \"cd345351-4fb6-41d2-b647-619bd9564d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"
Nov 02 10:25:05.740771 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.740746    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpvlt\" (UniqueName: \"kubernetes.io/projected/cd345351-4fb6-41d2-b647-619bd9564d13-kube-api-access-qpvlt\") pod \"maas-api-revoke-sub-d576e68b-j8wtk\" (UID: \"cd345351-4fb6-41d2-b647-619bd9564d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"
Nov 02 10:25:05.851625 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:05.851595    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"
Nov 02 10:25:06.130659 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:06.130630    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"]
Nov 02 10:25:06.131740 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:25:06.131715    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd345351_4fb6_41d2_b647_619bd9564d13.slice/crio-85ec76d304c71417216ca46159315817171b1096af6e1d768c659684ed54d21e WatchSource:0}: Error finding container 85ec76d304c71417216ca46159315817171b1096af6e1d768c659684ed54d21e: Status 404 returned error can't find the container with id 85ec76d304c71417216ca46159315817171b1096af6e1d768c659684ed54d21e
Nov 02 10:25:06.546629 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:25:06.546593    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:25:06.957622 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:06.957588    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk" event={"ID":"cd345351-4fb6-41d2-b647-619bd9564d13","Type":"ContainerStarted","Data":"7ff88d4ff295b59d48f5e211d7edaf5e56a71709c7922d2d96a74c5789cd580f"}
Nov 02 10:25:06.957846 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:06.957647    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk" event={"ID":"cd345351-4fb6-41d2-b647-619bd9564d13","Type":"ContainerStarted","Data":"85ec76d304c71417216ca46159315817171b1096af6e1d768c659684ed54d21e"}
Nov 02 10:25:06.977201 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:06.977148    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk" podStartSLOduration=1.977135188 podStartE2EDuration="1.977135188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:25:06.975927575 +0000 UTC m=+2983.914214355" watchObservedRunningTime="2026-11-02 10:25:06.977135188 +0000 UTC m=+2983.915421964"
Nov 02 10:25:07.965126 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:07.965086    4449 generic.go:356] "Generic (PLEG): container finished" podID="cd345351-4fb6-41d2-b647-619bd9564d13" containerID="7ff88d4ff295b59d48f5e211d7edaf5e56a71709c7922d2d96a74c5789cd580f" exitCode=0
Nov 02 10:25:07.965521 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:07.965179    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk" event={"ID":"cd345351-4fb6-41d2-b647-619bd9564d13","Type":"ContainerDied","Data":"7ff88d4ff295b59d48f5e211d7edaf5e56a71709c7922d2d96a74c5789cd580f"}
Nov 02 10:25:09.110584 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:09.110550    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"
Nov 02 10:25:09.158507 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:09.158472    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-xks-ca\") pod \"cd345351-4fb6-41d2-b647-619bd9564d13\" (UID: \"cd345351-4fb6-41d2-b647-619bd9564d13\") "
Nov 02 10:25:09.158687 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:09.158574    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd345351-4fb6-41d2-b647-619bd9564d13-kube-api-access-qpvlt\" (UniqueName: \"kubernetes.io/projected/cd345351-4fb6-41d2-b647-619bd9564d13-kube-api-access-qpvlt\") pod \"cd345351-4fb6-41d2-b647-619bd9564d13\" (UID: \"cd345351-4fb6-41d2-b647-619bd9564d13\") "
Nov 02 10:25:09.158735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:09.158691    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-openshift-ca\") pod \"cd345351-4fb6-41d2-b647-619bd9564d13\" (UID: \"cd345351-4fb6-41d2-b647-619bd9564d13\") "
Nov 02 10:25:09.159091 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:09.159059    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-openshift-ca" pod "cd345351-4fb6-41d2-b647-619bd9564d13" (UID: "cd345351-4fb6-41d2-b647-619bd9564d13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:25:09.160799 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:09.160773    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-xks-ca" pod "cd345351-4fb6-41d2-b647-619bd9564d13" (UID: "cd345351-4fb6-41d2-b647-619bd9564d13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:25:09.161043 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:09.161016    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd345351-4fb6-41d2-b647-619bd9564d13-kube-api-access-qpvlt" pod "cd345351-4fb6-41d2-b647-619bd9564d13" (UID: "cd345351-4fb6-41d2-b647-619bd9564d13"). InnerVolumeSpecName "kube-api-access-qpvlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:25:09.260296 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:09.260173    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:25:09.260296 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:09.260205    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpvlt\" (UniqueName: \"kubernetes.io/projected/cd345351-4fb6-41d2-b647-619bd9564d13-kube-api-access-qpvlt\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:25:09.260296 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:09.260248    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd345351-4fb6-41d2-b647-619bd9564d13-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:25:09.980134 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:09.980107    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"
Nov 02 10:25:09.980134 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:09.980111    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk" event={"ID":"cd345351-4fb6-41d2-b647-619bd9564d13","Type":"ContainerDied","Data":"85ec76d304c71417216ca46159315817171b1096af6e1d768c659684ed54d21e"}
Nov 02 10:25:09.980433 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:09.980159    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85ec76d304c71417216ca46159315817171b1096af6e1d768c659684ed54d21e"
Nov 02 10:25:21.816239 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:21.815305    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:25:21.816239 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:21.815387    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:25:22.917386 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:22.917351    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"]
Nov 02 10:25:22.917780 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:22.917773    4449 cpu_manager.go:414] "containerMap: removing container" podUID="cd345351-4fb6-41d2-b647-619bd9564d13" containerName="revoke-keys"
Nov 02 10:25:22.917826 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:22.917783    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd345351-4fb6-41d2-b647-619bd9564d13" containerName="revoke-keys"
Nov 02 10:25:22.917885 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:22.917875    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a6523a-4a18-412d-8b42-fe1e70daa302" containerName="istio-proxy"
Nov 02 10:25:22.917922 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:22.917888    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd345351-4fb6-41d2-b647-619bd9564d13" containerName="revoke-keys"
Nov 02 10:25:22.921114 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:22.921094    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"
Nov 02 10:25:22.927121 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:22.927095    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:25:22.928589 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:22.928553    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:25:22.928752 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:22.928621    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:25:22.928752 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:22.928685    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:25:22.931203 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:22.931173    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"]
Nov 02 10:25:23.043458 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:23.043418    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5bqlm\" (UID: \"2f524e42-88b7-4fed-9129-2f46a0832736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"
Nov 02 10:25:23.043651 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:23.043570    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5bqlm\" (UID: \"2f524e42-88b7-4fed-9129-2f46a0832736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"
Nov 02 10:25:23.043697 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:23.043651    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpvsx\" (UniqueName: \"kubernetes.io/projected/2f524e42-88b7-4fed-9129-2f46a0832736-kube-api-access-lpvsx\") pod \"maas-api-revoke-sub-d2bc93bc-5bqlm\" (UID: \"2f524e42-88b7-4fed-9129-2f46a0832736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"
Nov 02 10:25:23.145014 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:23.144973    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5bqlm\" (UID: \"2f524e42-88b7-4fed-9129-2f46a0832736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"
Nov 02 10:25:23.145197 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:23.145052    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpvsx\" (UniqueName: \"kubernetes.io/projected/2f524e42-88b7-4fed-9129-2f46a0832736-kube-api-access-lpvsx\") pod \"maas-api-revoke-sub-d2bc93bc-5bqlm\" (UID: \"2f524e42-88b7-4fed-9129-2f46a0832736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"
Nov 02 10:25:23.145197 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:23.145078    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5bqlm\" (UID: \"2f524e42-88b7-4fed-9129-2f46a0832736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"
Nov 02 10:25:23.145623 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:23.145602    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5bqlm\" (UID: \"2f524e42-88b7-4fed-9129-2f46a0832736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"
Nov 02 10:25:23.147670 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:23.147634    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5bqlm\" (UID: \"2f524e42-88b7-4fed-9129-2f46a0832736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"
Nov 02 10:25:23.158328 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:23.158305    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpvsx\" (UniqueName: \"kubernetes.io/projected/2f524e42-88b7-4fed-9129-2f46a0832736-kube-api-access-lpvsx\") pod \"maas-api-revoke-sub-d2bc93bc-5bqlm\" (UID: \"2f524e42-88b7-4fed-9129-2f46a0832736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"
Nov 02 10:25:23.236065 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:23.235983    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"
Nov 02 10:25:23.517419 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:23.517389    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"]
Nov 02 10:25:23.518737 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:25:23.518706    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f524e42_88b7_4fed_9129_2f46a0832736.slice/crio-3884f2bfa3d8037dddaba3fb5150903c2a1e0a39fee511ee284a662426f0951a WatchSource:0}: Error finding container 3884f2bfa3d8037dddaba3fb5150903c2a1e0a39fee511ee284a662426f0951a: Status 404 returned error can't find the container with id 3884f2bfa3d8037dddaba3fb5150903c2a1e0a39fee511ee284a662426f0951a
Nov 02 10:25:24.064083 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:24.064046    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm" event={"ID":"2f524e42-88b7-4fed-9129-2f46a0832736","Type":"ContainerStarted","Data":"6526234491b61630a3af9798764c8cbca1f873fcc51fac7e32cd9e9d09c41560"}
Nov 02 10:25:24.064508 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:24.064122    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm" event={"ID":"2f524e42-88b7-4fed-9129-2f46a0832736","Type":"ContainerStarted","Data":"3884f2bfa3d8037dddaba3fb5150903c2a1e0a39fee511ee284a662426f0951a"}
Nov 02 10:25:24.084824 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:24.084770    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm" podStartSLOduration=2.084757616 podStartE2EDuration="2.084757616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:25:24.08310141 +0000 UTC m=+3001.021388199" watchObservedRunningTime="2026-11-02 10:25:24.084757616 +0000 UTC m=+3001.023044428"
Nov 02 10:25:25.071194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:25.071161    4449 generic.go:356] "Generic (PLEG): container finished" podID="2f524e42-88b7-4fed-9129-2f46a0832736" containerID="6526234491b61630a3af9798764c8cbca1f873fcc51fac7e32cd9e9d09c41560" exitCode=0
Nov 02 10:25:25.071646 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:25.071256    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm" event={"ID":"2f524e42-88b7-4fed-9129-2f46a0832736","Type":"ContainerDied","Data":"6526234491b61630a3af9798764c8cbca1f873fcc51fac7e32cd9e9d09c41560"}
Nov 02 10:25:26.221159 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.221134    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"
Nov 02 10:25:26.377426 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.377322    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-openshift-ca\") pod \"2f524e42-88b7-4fed-9129-2f46a0832736\" (UID: \"2f524e42-88b7-4fed-9129-2f46a0832736\") "
Nov 02 10:25:26.377597 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.377479    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f524e42-88b7-4fed-9129-2f46a0832736-kube-api-access-lpvsx\" (UniqueName: \"kubernetes.io/projected/2f524e42-88b7-4fed-9129-2f46a0832736-kube-api-access-lpvsx\") pod \"2f524e42-88b7-4fed-9129-2f46a0832736\" (UID: \"2f524e42-88b7-4fed-9129-2f46a0832736\") "
Nov 02 10:25:26.377597 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.377505    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-xks-ca\") pod \"2f524e42-88b7-4fed-9129-2f46a0832736\" (UID: \"2f524e42-88b7-4fed-9129-2f46a0832736\") "
Nov 02 10:25:26.377787 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.377759    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-openshift-ca" pod "2f524e42-88b7-4fed-9129-2f46a0832736" (UID: "2f524e42-88b7-4fed-9129-2f46a0832736"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:25:26.379768 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.379735    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f524e42-88b7-4fed-9129-2f46a0832736-kube-api-access-lpvsx" pod "2f524e42-88b7-4fed-9129-2f46a0832736" (UID: "2f524e42-88b7-4fed-9129-2f46a0832736"). InnerVolumeSpecName "kube-api-access-lpvsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:25:26.379894 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.379797    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-xks-ca" pod "2f524e42-88b7-4fed-9129-2f46a0832736" (UID: "2f524e42-88b7-4fed-9129-2f46a0832736"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:25:26.478490 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.478455    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:25:26.478490 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.478483    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpvsx\" (UniqueName: \"kubernetes.io/projected/2f524e42-88b7-4fed-9129-2f46a0832736-kube-api-access-lpvsx\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:25:26.478490 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.478494    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f524e42-88b7-4fed-9129-2f46a0832736-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:25:26.707770 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.707675    4449 scope.go:122] "RemoveContainer" containerID="61c06da3d7b9969e11a904559610f25711e17955fd429ce317d621160a7a8089"
Nov 02 10:25:26.717921 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.717901    4449 scope.go:122] "RemoveContainer" containerID="3dbb82abd54a60e615189971e338b5368e3e6bbf5e49302f284733f16dc41bb8"
Nov 02 10:25:26.726851 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.726830    4449 scope.go:122] "RemoveContainer" containerID="5456388dd3212cb42d3366f667f66bfee1fd90619a2bf279e7af885eee046d6b"
Nov 02 10:25:26.735489 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:26.735470    4449 scope.go:122] "RemoveContainer" containerID="f6eb3a5640b5233f36c156a509640e2acbe9e155251eaadab3ffe5057bfbe7f3"
Nov 02 10:25:27.084798 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:27.084765    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"
Nov 02 10:25:27.084798 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:27.084786    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm" event={"ID":"2f524e42-88b7-4fed-9129-2f46a0832736","Type":"ContainerDied","Data":"3884f2bfa3d8037dddaba3fb5150903c2a1e0a39fee511ee284a662426f0951a"}
Nov 02 10:25:27.085016 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:27.084834    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3884f2bfa3d8037dddaba3fb5150903c2a1e0a39fee511ee284a662426f0951a"
Nov 02 10:25:36.546871 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:25:36.546786    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:25:43.391586 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.391546    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"]
Nov 02 10:25:43.392198 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.392177    4449 cpu_manager.go:414] "containerMap: removing container" podUID="2f524e42-88b7-4fed-9129-2f46a0832736" containerName="revoke-keys"
Nov 02 10:25:43.392331 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.392202    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f524e42-88b7-4fed-9129-2f46a0832736" containerName="revoke-keys"
Nov 02 10:25:43.392423 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.392409    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f524e42-88b7-4fed-9129-2f46a0832736" containerName="revoke-keys"
Nov 02 10:25:43.397198 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.397177    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"
Nov 02 10:25:43.401557 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.401532    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:25:43.402353 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.401676    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:25:43.402353 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.402185    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:25:43.402531 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.402367    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:25:43.404501 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.404474    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"]
Nov 02 10:25:43.460130 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.460097    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637caa1a-dv2pz\" (UID: \"84702b08-4b18-4147-86c0-f554c82db50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"
Nov 02 10:25:43.460330 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.460144    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637caa1a-dv2pz\" (UID: \"84702b08-4b18-4147-86c0-f554c82db50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"
Nov 02 10:25:43.460330 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.460265    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp222\" (UniqueName: \"kubernetes.io/projected/84702b08-4b18-4147-86c0-f554c82db50d-kube-api-access-xp222\") pod \"maas-api-revoke-sub-637caa1a-dv2pz\" (UID: \"84702b08-4b18-4147-86c0-f554c82db50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"
Nov 02 10:25:43.561012 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.560975    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637caa1a-dv2pz\" (UID: \"84702b08-4b18-4147-86c0-f554c82db50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"
Nov 02 10:25:43.561281 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.561042    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp222\" (UniqueName: \"kubernetes.io/projected/84702b08-4b18-4147-86c0-f554c82db50d-kube-api-access-xp222\") pod \"maas-api-revoke-sub-637caa1a-dv2pz\" (UID: \"84702b08-4b18-4147-86c0-f554c82db50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"
Nov 02 10:25:43.561281 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.561119    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637caa1a-dv2pz\" (UID: \"84702b08-4b18-4147-86c0-f554c82db50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"
Nov 02 10:25:43.561845 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.561824    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637caa1a-dv2pz\" (UID: \"84702b08-4b18-4147-86c0-f554c82db50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"
Nov 02 10:25:43.563650 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.563627    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637caa1a-dv2pz\" (UID: \"84702b08-4b18-4147-86c0-f554c82db50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"
Nov 02 10:25:43.574366 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.574345    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp222\" (UniqueName: \"kubernetes.io/projected/84702b08-4b18-4147-86c0-f554c82db50d-kube-api-access-xp222\") pod \"maas-api-revoke-sub-637caa1a-dv2pz\" (UID: \"84702b08-4b18-4147-86c0-f554c82db50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"
Nov 02 10:25:43.715380 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:43.715288    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"
Nov 02 10:25:44.008239 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:44.008137    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"]
Nov 02 10:25:44.013666 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:25:44.013630    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84702b08_4b18_4147_86c0_f554c82db50d.slice/crio-85b361c5ca94e9df902173838bbd7e9f8b1b505e5db03825d34714a9a8657747 WatchSource:0}: Error finding container 85b361c5ca94e9df902173838bbd7e9f8b1b505e5db03825d34714a9a8657747: Status 404 returned error can't find the container with id 85b361c5ca94e9df902173838bbd7e9f8b1b505e5db03825d34714a9a8657747
Nov 02 10:25:44.192753 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:44.192709    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz" event={"ID":"84702b08-4b18-4147-86c0-f554c82db50d","Type":"ContainerStarted","Data":"85b361c5ca94e9df902173838bbd7e9f8b1b505e5db03825d34714a9a8657747"}
Nov 02 10:25:45.230097 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:45.230000    4449 generic.go:356] "Generic (PLEG): container finished" podID="84702b08-4b18-4147-86c0-f554c82db50d" containerID="709b33a530b08f48e90b3c8eeb9479f16c46488c6e57d1a0ae3f861b59d633a7" exitCode=0
Nov 02 10:25:45.230097 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:45.230064    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz" event={"ID":"84702b08-4b18-4147-86c0-f554c82db50d","Type":"ContainerDied","Data":"709b33a530b08f48e90b3c8eeb9479f16c46488c6e57d1a0ae3f861b59d633a7"}
Nov 02 10:25:46.028362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:46.028280    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"]
Nov 02 10:25:46.030063 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:46.030037    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-bx4kv"]
Nov 02 10:25:46.617280 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:46.616511    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"
Nov 02 10:25:46.736293 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:46.736188    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-openshift-ca\") pod \"84702b08-4b18-4147-86c0-f554c82db50d\" (UID: \"84702b08-4b18-4147-86c0-f554c82db50d\") "
Nov 02 10:25:46.736293 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:46.736286    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84702b08-4b18-4147-86c0-f554c82db50d-kube-api-access-xp222\" (UniqueName: \"kubernetes.io/projected/84702b08-4b18-4147-86c0-f554c82db50d-kube-api-access-xp222\") pod \"84702b08-4b18-4147-86c0-f554c82db50d\" (UID: \"84702b08-4b18-4147-86c0-f554c82db50d\") "
Nov 02 10:25:46.736507 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:46.736394    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-xks-ca\") pod \"84702b08-4b18-4147-86c0-f554c82db50d\" (UID: \"84702b08-4b18-4147-86c0-f554c82db50d\") "
Nov 02 10:25:46.737161 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:46.737122    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-openshift-ca" pod "84702b08-4b18-4147-86c0-f554c82db50d" (UID: "84702b08-4b18-4147-86c0-f554c82db50d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:25:46.738806 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:46.738763    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-xks-ca" pod "84702b08-4b18-4147-86c0-f554c82db50d" (UID: "84702b08-4b18-4147-86c0-f554c82db50d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:25:46.739088 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:46.739070    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84702b08-4b18-4147-86c0-f554c82db50d-kube-api-access-xp222" pod "84702b08-4b18-4147-86c0-f554c82db50d" (UID: "84702b08-4b18-4147-86c0-f554c82db50d"). InnerVolumeSpecName "kube-api-access-xp222". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:25:46.844754 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:46.838662    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:25:46.844754 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:46.838720    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp222\" (UniqueName: \"kubernetes.io/projected/84702b08-4b18-4147-86c0-f554c82db50d-kube-api-access-xp222\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:25:46.844754 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:46.838740    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84702b08-4b18-4147-86c0-f554c82db50d-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:25:47.219242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:47.218091    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8a7cb9-325d-4203-8bac-a2d88fdb9b31" path="/var/lib/kubelet/pods/8b8a7cb9-325d-4203-8bac-a2d88fdb9b31/volumes"
Nov 02 10:25:47.243204 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:47.243180    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"
Nov 02 10:25:47.243363 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:47.243211    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz" event={"ID":"84702b08-4b18-4147-86c0-f554c82db50d","Type":"ContainerDied","Data":"85b361c5ca94e9df902173838bbd7e9f8b1b505e5db03825d34714a9a8657747"}
Nov 02 10:25:47.243363 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:47.243268    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85b361c5ca94e9df902173838bbd7e9f8b1b505e5db03825d34714a9a8657747"
Nov 02 10:25:56.026061 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:56.026022    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"]
Nov 02 10:25:56.028503 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:56.028481    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4spmr"]
Nov 02 10:25:57.205789 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:25:57.205752    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a90313-de8e-42e0-aeba-24037f530cc5" path="/var/lib/kubelet/pods/02a90313-de8e-42e0-aeba-24037f530cc5/volumes"
Nov 02 10:26:06.544682 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:26:06.544648    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:26:26.769777 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:26:26.769747    4449 scope.go:122] "RemoveContainer" containerID="dfbc2a06267d290f1d6516073a357f0b3bf394250749333d461c505aec172196"
Nov 02 10:26:26.779127 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:26:26.779107    4449 scope.go:122] "RemoveContainer" containerID="8e3f3e3d7aa16553bc9f3a81c75b01becde340920eb629d7430a312e35a8da62"
Nov 02 10:26:26.787493 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:26:26.787476    4449 scope.go:122] "RemoveContainer" containerID="802b63f15a506e6763c79952161ac4a3f6c0422a00fc0fb6702dec61c289174b"
Nov 02 10:26:26.795561 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:26:26.795540    4449 scope.go:122] "RemoveContainer" containerID="cc54b965c8d1147315147603b01f80b5c48b06b6c43f12ed3cf2ab3e9bcc6345"
Nov 02 10:26:36.545113 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:26:36.545080    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:27:06.544993 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:27:06.544920    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:27:18.026004 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:27:18.025968    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"]
Nov 02 10:27:18.031747 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:27:18.031723    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1bf7e6b-7rgfb"]
Nov 02 10:27:19.204473 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:27:19.204434    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08d3743d-cb44-40bd-9844-19a2b82eec63" path="/var/lib/kubelet/pods/08d3743d-cb44-40bd-9844-19a2b82eec63/volumes"
Nov 02 10:27:26.830427 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:27:26.830393    4449 scope.go:122] "RemoveContainer" containerID="cf8df310c594f5f9c2138c66a5bf834b0885b7f54f8cbe472b85eb828a359e0e"
Nov 02 10:27:26.839732 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:27:26.839712    4449 scope.go:122] "RemoveContainer" containerID="f6d455246ca79a2ac9816d60ae6568186bad3e720f269a1c66a9cbc2411a5b6e"
Nov 02 10:27:26.848449 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:27:26.848431    4449 scope.go:122] "RemoveContainer" containerID="6620970cf4a24cd58d77ea5e3bfc6d92b32dba25896004b74ede80837de93f2d"
Nov 02 10:27:36.545024 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:27:36.544993    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:28:06.544166 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:28:06.544131    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:28:26.876772 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:28:26.876742    4449 scope.go:122] "RemoveContainer" containerID="c91ef0659e5a4e6c52038ffcc8f15d102e866190c5c22cec94482412d0bf6ac3"
Nov 02 10:28:30.024867 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:28:30.024832    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"]
Nov 02 10:28:30.029830 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:28:30.029807    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6r776"]
Nov 02 10:28:31.204838 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:28:31.204808    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db040ad0-591e-409c-80cd-83d594857877" path="/var/lib/kubelet/pods/db040ad0-591e-409c-80cd-83d594857877/volumes"
Nov 02 10:28:36.544832 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:28:36.544759    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:29:06.545172 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:29:06.545138    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:29:26.899095 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:29:26.899066    4449 scope.go:122] "RemoveContainer" containerID="fd9207633574caea0ddd46be861eee7245d75fc69f6ed59e151e36b0ecdf9f70"
Nov 02 10:29:36.544425 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:29:36.544391    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:30:00.153074 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.153044    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"]
Nov 02 10:30:00.153609 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.153448    4449 cpu_manager.go:414] "containerMap: removing container" podUID="84702b08-4b18-4147-86c0-f554c82db50d" containerName="revoke-keys"
Nov 02 10:30:00.153609 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.153460    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84702b08-4b18-4147-86c0-f554c82db50d" containerName="revoke-keys"
Nov 02 10:30:00.153609 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.153562    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="84702b08-4b18-4147-86c0-f554c82db50d" containerName="revoke-keys"
Nov 02 10:30:00.156622 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.156607    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"
Nov 02 10:30:00.161671 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.161647    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:30:00.161792 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.161651    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:30:00.161792 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.161704    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:30:00.174055 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.174033    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"]
Nov 02 10:30:00.224630 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.224600    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjtcb\" (UniqueName: \"kubernetes.io/projected/3f245b17-d58a-4a81-ab93-b324f817fc28-kube-api-access-zjtcb\") pod \"maas-api-key-cleanup-29893590-nz5k2\" (UID: \"3f245b17-d58a-4a81-ab93-b324f817fc28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"
Nov 02 10:30:00.224758 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.224640    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f245b17-d58a-4a81-ab93-b324f817fc28-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893590-nz5k2\" (UID: \"3f245b17-d58a-4a81-ab93-b324f817fc28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"
Nov 02 10:30:00.325335 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.325310    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjtcb\" (UniqueName: \"kubernetes.io/projected/3f245b17-d58a-4a81-ab93-b324f817fc28-kube-api-access-zjtcb\") pod \"maas-api-key-cleanup-29893590-nz5k2\" (UID: \"3f245b17-d58a-4a81-ab93-b324f817fc28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"
Nov 02 10:30:00.325436 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.325343    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f245b17-d58a-4a81-ab93-b324f817fc28-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893590-nz5k2\" (UID: \"3f245b17-d58a-4a81-ab93-b324f817fc28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"
Nov 02 10:30:00.325925 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.325907    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f245b17-d58a-4a81-ab93-b324f817fc28-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893590-nz5k2\" (UID: \"3f245b17-d58a-4a81-ab93-b324f817fc28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"
Nov 02 10:30:00.338370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.338345    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjtcb\" (UniqueName: \"kubernetes.io/projected/3f245b17-d58a-4a81-ab93-b324f817fc28-kube-api-access-zjtcb\") pod \"maas-api-key-cleanup-29893590-nz5k2\" (UID: \"3f245b17-d58a-4a81-ab93-b324f817fc28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"
Nov 02 10:30:00.469931 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.469880    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"
Nov 02 10:30:00.742710 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:00.742683    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"]
Nov 02 10:30:00.743804 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:30:00.743777    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f245b17_d58a_4a81_ab93_b324f817fc28.slice/crio-622a40a847a207dd618a802ae5197bf8cc56d2c39fba2785af98e77aa8eb452e WatchSource:0}: Error finding container 622a40a847a207dd618a802ae5197bf8cc56d2c39fba2785af98e77aa8eb452e: Status 404 returned error can't find the container with id 622a40a847a207dd618a802ae5197bf8cc56d2c39fba2785af98e77aa8eb452e
Nov 02 10:30:01.606221 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:01.606192    4449 generic.go:356] "Generic (PLEG): container finished" podID="3f245b17-d58a-4a81-ab93-b324f817fc28" containerID="b3acab063d425877376e10a36ecf75b51242a580096dd6407f780b04f78666c3" exitCode=60
Nov 02 10:30:01.606583 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:01.606274    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2" event={"ID":"3f245b17-d58a-4a81-ab93-b324f817fc28","Type":"ContainerDied","Data":"b3acab063d425877376e10a36ecf75b51242a580096dd6407f780b04f78666c3"}
Nov 02 10:30:01.606583 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:01.606345    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2" event={"ID":"3f245b17-d58a-4a81-ab93-b324f817fc28","Type":"ContainerStarted","Data":"622a40a847a207dd618a802ae5197bf8cc56d2c39fba2785af98e77aa8eb452e"}
Nov 02 10:30:01.606583 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:01.606498    4449 scope.go:122] "RemoveContainer" containerID="b3acab063d425877376e10a36ecf75b51242a580096dd6407f780b04f78666c3"
Nov 02 10:30:02.613797 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:02.613771    4449 generic.go:356] "Generic (PLEG): container finished" podID="3f245b17-d58a-4a81-ab93-b324f817fc28" containerID="17d5a81bfb3175ee55755bdc403209c709dbd0ffb96f25a39ed05922c41b31a0" exitCode=60
Nov 02 10:30:02.614198 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:02.613853    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2" event={"ID":"3f245b17-d58a-4a81-ab93-b324f817fc28","Type":"ContainerDied","Data":"17d5a81bfb3175ee55755bdc403209c709dbd0ffb96f25a39ed05922c41b31a0"}
Nov 02 10:30:02.614198 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:02.613906    4449 scope.go:122] "RemoveContainer" containerID="b3acab063d425877376e10a36ecf75b51242a580096dd6407f780b04f78666c3"
Nov 02 10:30:02.614198 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:02.614079    4449 scope.go:122] "RemoveContainer" containerID="17d5a81bfb3175ee55755bdc403209c709dbd0ffb96f25a39ed05922c41b31a0"
Nov 02 10:30:02.614407 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:30:02.614317    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893590-nz5k2_odh-ai-gateway-infra(3f245b17-d58a-4a81-ab93-b324f817fc28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2" podUID="3f245b17-d58a-4a81-ab93-b324f817fc28"
Nov 02 10:30:03.620261 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:03.620238    4449 scope.go:122] "RemoveContainer" containerID="17d5a81bfb3175ee55755bdc403209c709dbd0ffb96f25a39ed05922c41b31a0"
Nov 02 10:30:03.620651 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:30:03.620412    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893590-nz5k2_odh-ai-gateway-infra(3f245b17-d58a-4a81-ab93-b324f817fc28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2" podUID="3f245b17-d58a-4a81-ab93-b324f817fc28"
Nov 02 10:30:06.545504 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:30:06.545428    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:30:10.034137 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:10.034109    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"]
Nov 02 10:30:10.036297 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:10.036278    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j8wtk"]
Nov 02 10:30:11.204380 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:11.204352    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd345351-4fb6-41d2-b647-619bd9564d13" path="/var/lib/kubelet/pods/cd345351-4fb6-41d2-b647-619bd9564d13/volumes"
Nov 02 10:30:13.201965 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:13.201940    4449 scope.go:122] "RemoveContainer" containerID="17d5a81bfb3175ee55755bdc403209c709dbd0ffb96f25a39ed05922c41b31a0"
Nov 02 10:30:13.681019 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:13.680993    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2" event={"ID":"3f245b17-d58a-4a81-ab93-b324f817fc28","Type":"ContainerStarted","Data":"7b1ebe794962acc8dc46c0691ca9c1d6487e8b27a5de9d4bcb3d76db3fe25115"}
Nov 02 10:30:13.699809 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:13.699768    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2" podStartSLOduration=13.69975591 podStartE2EDuration="13.69975591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:30:13.697840514 +0000 UTC m=+3290.636127302" watchObservedRunningTime="2026-11-02 10:30:13.69975591 +0000 UTC m=+3290.638042712"
Nov 02 10:30:14.687605 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:14.687583    4449 generic.go:356] "Generic (PLEG): container finished" podID="3f245b17-d58a-4a81-ab93-b324f817fc28" containerID="7b1ebe794962acc8dc46c0691ca9c1d6487e8b27a5de9d4bcb3d76db3fe25115" exitCode=60
Nov 02 10:30:14.687978 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:14.687649    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2" event={"ID":"3f245b17-d58a-4a81-ab93-b324f817fc28","Type":"ContainerDied","Data":"7b1ebe794962acc8dc46c0691ca9c1d6487e8b27a5de9d4bcb3d76db3fe25115"}
Nov 02 10:30:14.687978 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:14.687703    4449 scope.go:122] "RemoveContainer" containerID="17d5a81bfb3175ee55755bdc403209c709dbd0ffb96f25a39ed05922c41b31a0"
Nov 02 10:30:14.687978 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:14.687944    4449 scope.go:122] "RemoveContainer" containerID="7b1ebe794962acc8dc46c0691ca9c1d6487e8b27a5de9d4bcb3d76db3fe25115"
Nov 02 10:30:14.688141 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:30:14.688120    4449 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893590-nz5k2_odh-ai-gateway-infra(3f245b17-d58a-4a81-ab93-b324f817fc28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2" podUID="3f245b17-d58a-4a81-ab93-b324f817fc28"
Nov 02 10:30:14.709787 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:14.709765    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"]
Nov 02 10:30:15.830783 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:15.830761    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"
Nov 02 10:30:15.953178 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:15.953099    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f245b17-d58a-4a81-ab93-b324f817fc28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f245b17-d58a-4a81-ab93-b324f817fc28-maas-api-service-ca\") pod \"3f245b17-d58a-4a81-ab93-b324f817fc28\" (UID: \"3f245b17-d58a-4a81-ab93-b324f817fc28\") "
Nov 02 10:30:15.953372 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:15.953202    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f245b17-d58a-4a81-ab93-b324f817fc28-kube-api-access-zjtcb\" (UniqueName: \"kubernetes.io/projected/3f245b17-d58a-4a81-ab93-b324f817fc28-kube-api-access-zjtcb\") pod \"3f245b17-d58a-4a81-ab93-b324f817fc28\" (UID: \"3f245b17-d58a-4a81-ab93-b324f817fc28\") "
Nov 02 10:30:15.953458 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:15.953432    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f245b17-d58a-4a81-ab93-b324f817fc28-maas-api-service-ca" pod "3f245b17-d58a-4a81-ab93-b324f817fc28" (UID: "3f245b17-d58a-4a81-ab93-b324f817fc28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:30:15.953585 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:15.953572    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f245b17-d58a-4a81-ab93-b324f817fc28-maas-api-service-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:30:15.955198 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:15.955177    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f245b17-d58a-4a81-ab93-b324f817fc28-kube-api-access-zjtcb" pod "3f245b17-d58a-4a81-ab93-b324f817fc28" (UID: "3f245b17-d58a-4a81-ab93-b324f817fc28"). InnerVolumeSpecName "kube-api-access-zjtcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:30:16.054982 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:16.054961    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjtcb\" (UniqueName: \"kubernetes.io/projected/3f245b17-d58a-4a81-ab93-b324f817fc28-kube-api-access-zjtcb\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:30:16.700898 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:16.700879    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"
Nov 02 10:30:16.701008 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:16.700922    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2" event={"ID":"3f245b17-d58a-4a81-ab93-b324f817fc28","Type":"ContainerDied","Data":"622a40a847a207dd618a802ae5197bf8cc56d2c39fba2785af98e77aa8eb452e"}
Nov 02 10:30:16.701008 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:16.700972    4449 scope.go:122] "RemoveContainer" containerID="7b1ebe794962acc8dc46c0691ca9c1d6487e8b27a5de9d4bcb3d76db3fe25115"
Nov 02 10:30:16.723664 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:16.723638    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"]
Nov 02 10:30:16.730401 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:16.730379    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893590-nz5k2"]
Nov 02 10:30:17.204529 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:17.204504    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f245b17-d58a-4a81-ab93-b324f817fc28" path="/var/lib/kubelet/pods/3f245b17-d58a-4a81-ab93-b324f817fc28/volumes"
Nov 02 10:30:26.920061 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:26.920031    4449 scope.go:122] "RemoveContainer" containerID="eb92d0cf086a3f0d55ec6fccc79ff3259975c0153fba61643b61bf3f8ac0177a"
Nov 02 10:30:27.027575 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:27.027551    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"]
Nov 02 10:30:27.031684 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:27.031660    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5bqlm"]
Nov 02 10:30:27.205025 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:27.204967    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f524e42-88b7-4fed-9129-2f46a0832736" path="/var/lib/kubelet/pods/2f524e42-88b7-4fed-9129-2f46a0832736/volumes"
Nov 02 10:30:36.544025 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:30:36.543986    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:30:47.025018 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:47.024984    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"]
Nov 02 10:30:47.027780 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:47.027753    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637caa1a-dv2pz"]
Nov 02 10:30:47.204633 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:30:47.204598    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84702b08-4b18-4147-86c0-f554c82db50d" path="/var/lib/kubelet/pods/84702b08-4b18-4147-86c0-f554c82db50d/volumes"
Nov 02 10:31:06.546812 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:31:06.546780    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:31:11.225839 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:11.225802    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"]
Nov 02 10:31:11.226339 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:11.225966    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy" containerID="cri-o://ba9e4e41a821070bd7eac46538180d5ed5740a4de6b3ff439766252b1a1682d4" gracePeriod=30
Nov 02 10:31:16.473997 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.473974    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 10:31:16.553505 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.553476    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-credential-socket\") pod \"98145c71-b388-4093-83be-cf5f18cf1af0\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") "
Nov 02 10:31:16.553692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.553516    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-istio-token\" (UniqueName: \"kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-istio-token\") pod \"98145c71-b388-4093-83be-cf5f18cf1af0\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") "
Nov 02 10:31:16.553692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.553551    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-kube-api-access-xxfng\" (UniqueName: \"kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-kube-api-access-xxfng\") pod \"98145c71-b388-4093-83be-cf5f18cf1af0\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") "
Nov 02 10:31:16.553692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.553590    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-certs\") pod \"98145c71-b388-4093-83be-cf5f18cf1af0\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") "
Nov 02 10:31:16.553882 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.553749    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98145c71-b388-4093-83be-cf5f18cf1af0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98145c71-b388-4093-83be-cf5f18cf1af0-istio-podinfo\") pod \"98145c71-b388-4093-83be-cf5f18cf1af0\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") "
Nov 02 10:31:16.553882 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.553782    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-credential-socket" pod "98145c71-b388-4093-83be-cf5f18cf1af0" (UID: "98145c71-b388-4093-83be-cf5f18cf1af0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.553882 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.553802    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-envoy\") pod \"98145c71-b388-4093-83be-cf5f18cf1af0\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") "
Nov 02 10:31:16.553882 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.553851    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-data\") pod \"98145c71-b388-4093-83be-cf5f18cf1af0\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") "
Nov 02 10:31:16.554277 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.553893    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98145c71-b388-4093-83be-cf5f18cf1af0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98145c71-b388-4093-83be-cf5f18cf1af0-istiod-ca-cert\") pod \"98145c71-b388-4093-83be-cf5f18cf1af0\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") "
Nov 02 10:31:16.554277 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.553903    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-certs" pod "98145c71-b388-4093-83be-cf5f18cf1af0" (UID: "98145c71-b388-4093-83be-cf5f18cf1af0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.554277 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.553972    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-socket\") pod \"98145c71-b388-4093-83be-cf5f18cf1af0\" (UID: \"98145c71-b388-4093-83be-cf5f18cf1af0\") "
Nov 02 10:31:16.554277 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.554106    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-data" pod "98145c71-b388-4093-83be-cf5f18cf1af0" (UID: "98145c71-b388-4093-83be-cf5f18cf1af0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.554654 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.554337    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-socket" pod "98145c71-b388-4093-83be-cf5f18cf1af0" (UID: "98145c71-b388-4093-83be-cf5f18cf1af0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.554654 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.554359    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98145c71-b388-4093-83be-cf5f18cf1af0-istiod-ca-cert" pod "98145c71-b388-4093-83be-cf5f18cf1af0" (UID: "98145c71-b388-4093-83be-cf5f18cf1af0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:31:16.554654 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.554576    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.554654 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.554596    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.554654 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.554611    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.554654 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.554626    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98145c71-b388-4093-83be-cf5f18cf1af0-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.554654 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.554641    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.556096 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.556065    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-envoy" pod "98145c71-b388-4093-83be-cf5f18cf1af0" (UID: "98145c71-b388-4093-83be-cf5f18cf1af0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.556192 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.556095    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-kube-api-access-xxfng" pod "98145c71-b388-4093-83be-cf5f18cf1af0" (UID: "98145c71-b388-4093-83be-cf5f18cf1af0"). InnerVolumeSpecName "kube-api-access-xxfng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:31:16.556291 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.556262    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98145c71-b388-4093-83be-cf5f18cf1af0-istio-podinfo" pod "98145c71-b388-4093-83be-cf5f18cf1af0" (UID: "98145c71-b388-4093-83be-cf5f18cf1af0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:31:16.556641 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.556618    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-istio-token" pod "98145c71-b388-4093-83be-cf5f18cf1af0" (UID: "98145c71-b388-4093-83be-cf5f18cf1af0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:31:16.655491 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.655462    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.655605 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.655497    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxfng\" (UniqueName: \"kubernetes.io/projected/98145c71-b388-4093-83be-cf5f18cf1af0-kube-api-access-xxfng\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.655605 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.655515    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98145c71-b388-4093-83be-cf5f18cf1af0-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.655605 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:16.655529    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98145c71-b388-4093-83be-cf5f18cf1af0-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:17.012626 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:17.012553    4449 generic.go:356] "Generic (PLEG): container finished" podID="98145c71-b388-4093-83be-cf5f18cf1af0" containerID="ba9e4e41a821070bd7eac46538180d5ed5740a4de6b3ff439766252b1a1682d4" exitCode=0
Nov 02 10:31:17.012626 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:17.012602    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"
Nov 02 10:31:17.012791 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:17.012628    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" event={"ID":"98145c71-b388-4093-83be-cf5f18cf1af0","Type":"ContainerDied","Data":"ba9e4e41a821070bd7eac46538180d5ed5740a4de6b3ff439766252b1a1682d4"}
Nov 02 10:31:17.012791 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:17.012680    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk" event={"ID":"98145c71-b388-4093-83be-cf5f18cf1af0","Type":"ContainerDied","Data":"c5fe06a2d1783679db3a80c8af3c18d4af62aadfc7bde3de69a31e056ac0c15f"}
Nov 02 10:31:17.012791 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:17.012701    4449 scope.go:122] "RemoveContainer" containerID="ba9e4e41a821070bd7eac46538180d5ed5740a4de6b3ff439766252b1a1682d4"
Nov 02 10:31:17.022839 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:17.022824    4449 scope.go:122] "RemoveContainer" containerID="ba9e4e41a821070bd7eac46538180d5ed5740a4de6b3ff439766252b1a1682d4"
Nov 02 10:31:17.023094 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:31:17.023079    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9e4e41a821070bd7eac46538180d5ed5740a4de6b3ff439766252b1a1682d4\": container with ID starting with ba9e4e41a821070bd7eac46538180d5ed5740a4de6b3ff439766252b1a1682d4 not found: ID does not exist" containerID="ba9e4e41a821070bd7eac46538180d5ed5740a4de6b3ff439766252b1a1682d4"
Nov 02 10:31:17.023151 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:17.023100    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9e4e41a821070bd7eac46538180d5ed5740a4de6b3ff439766252b1a1682d4"} err="failed to get container status \"ba9e4e41a821070bd7eac46538180d5ed5740a4de6b3ff439766252b1a1682d4\": rpc error: code = NotFound desc = could not find container \"ba9e4e41a821070bd7eac46538180d5ed5740a4de6b3ff439766252b1a1682d4\": container with ID starting with ba9e4e41a821070bd7eac46538180d5ed5740a4de6b3ff439766252b1a1682d4 not found: ID does not exist"
Nov 02 10:31:17.037958 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:17.037933    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"]
Nov 02 10:31:17.042947 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:17.042928    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7cnfk"]
Nov 02 10:31:17.204852 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:17.204830    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" path="/var/lib/kubelet/pods/98145c71-b388-4093-83be-cf5f18cf1af0/volumes"
Nov 02 10:31:26.246421 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:26.246391    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"]
Nov 02 10:31:26.246917 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:26.246570    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy" containerID="cri-o://5ef7cb2e4084b0be9cbca2bb936b5641639d23f375f9f23660d8ff6a34a42ad4" gracePeriod=30
Nov 02 10:31:26.944866 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:26.944839    4449 scope.go:122] "RemoveContainer" containerID="6526234491b61630a3af9798764c8cbca1f873fcc51fac7e32cd9e9d09c41560"
Nov 02 10:31:26.953501 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:26.953477    4449 scope.go:122] "RemoveContainer" containerID="7ff88d4ff295b59d48f5e211d7edaf5e56a71709c7922d2d96a74c5789cd580f"
Nov 02 10:31:31.485272 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.485251    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 10:31:31.581803 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.581768    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-kube-api-access-ntchs\" (UniqueName: \"kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-kube-api-access-ntchs\") pod \"e5c98f31-5358-46e5-825b-ab24b569f10b\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") "
Nov 02 10:31:31.581951 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.581810    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-envoy\") pod \"e5c98f31-5358-46e5-825b-ab24b569f10b\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") "
Nov 02 10:31:31.581951 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.581830    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-data\") pod \"e5c98f31-5358-46e5-825b-ab24b569f10b\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") "
Nov 02 10:31:31.581951 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.581858    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-certs\") pod \"e5c98f31-5358-46e5-825b-ab24b569f10b\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") "
Nov 02 10:31:31.581951 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.581884    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-podinfo\") pod \"e5c98f31-5358-46e5-825b-ab24b569f10b\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") "
Nov 02 10:31:31.581951 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.581910    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c98f31-5358-46e5-825b-ab24b569f10b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c98f31-5358-46e5-825b-ab24b569f10b-istiod-ca-cert\") pod \"e5c98f31-5358-46e5-825b-ab24b569f10b\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") "
Nov 02 10:31:31.582360 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.582059    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-token\" (UniqueName: \"kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-token\") pod \"e5c98f31-5358-46e5-825b-ab24b569f10b\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") "
Nov 02 10:31:31.582360 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.582132    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-credential-socket\") pod \"e5c98f31-5358-46e5-825b-ab24b569f10b\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") "
Nov 02 10:31:31.582360 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.582159    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-socket\") pod \"e5c98f31-5358-46e5-825b-ab24b569f10b\" (UID: \"e5c98f31-5358-46e5-825b-ab24b569f10b\") "
Nov 02 10:31:31.582360 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.582268    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-data" pod "e5c98f31-5358-46e5-825b-ab24b569f10b" (UID: "e5c98f31-5358-46e5-825b-ab24b569f10b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:31.582360 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.582347    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c98f31-5358-46e5-825b-ab24b569f10b-istiod-ca-cert" pod "e5c98f31-5358-46e5-825b-ab24b569f10b" (UID: "e5c98f31-5358-46e5-825b-ab24b569f10b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:31:31.582838 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.582482    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-credential-socket" pod "e5c98f31-5358-46e5-825b-ab24b569f10b" (UID: "e5c98f31-5358-46e5-825b-ab24b569f10b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:31.582838 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.582552    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-socket" pod "e5c98f31-5358-46e5-825b-ab24b569f10b" (UID: "e5c98f31-5358-46e5-825b-ab24b569f10b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:31.582838 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.582660    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-certs" pod "e5c98f31-5358-46e5-825b-ab24b569f10b" (UID: "e5c98f31-5358-46e5-825b-ab24b569f10b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:31.583007 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.582894    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:31.583007 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.582916    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:31.583007 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.582931    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:31.583007 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.582945    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:31.583007 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.582957    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c98f31-5358-46e5-825b-ab24b569f10b-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:31.584379 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.584349    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-kube-api-access-ntchs" pod "e5c98f31-5358-46e5-825b-ab24b569f10b" (UID: "e5c98f31-5358-46e5-825b-ab24b569f10b"). InnerVolumeSpecName "kube-api-access-ntchs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:31:31.584633 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.584611    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-envoy" pod "e5c98f31-5358-46e5-825b-ab24b569f10b" (UID: "e5c98f31-5358-46e5-825b-ab24b569f10b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:31.584685 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.584614    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-podinfo" pod "e5c98f31-5358-46e5-825b-ab24b569f10b" (UID: "e5c98f31-5358-46e5-825b-ab24b569f10b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:31:31.584765 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.584747    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-token" pod "e5c98f31-5358-46e5-825b-ab24b569f10b" (UID: "e5c98f31-5358-46e5-825b-ab24b569f10b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:31:31.683787 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.683765    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:31.683880 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.683789    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntchs\" (UniqueName: \"kubernetes.io/projected/e5c98f31-5358-46e5-825b-ab24b569f10b-kube-api-access-ntchs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:31.683880 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.683799    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:31.683880 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:31.683807    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c98f31-5358-46e5-825b-ab24b569f10b-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:32.099934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:32.099912    4449 generic.go:356] "Generic (PLEG): container finished" podID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerID="5ef7cb2e4084b0be9cbca2bb936b5641639d23f375f9f23660d8ff6a34a42ad4" exitCode=0
Nov 02 10:31:32.100118 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:32.099958    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"
Nov 02 10:31:32.100118 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:32.099967    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" event={"ID":"e5c98f31-5358-46e5-825b-ab24b569f10b","Type":"ContainerDied","Data":"5ef7cb2e4084b0be9cbca2bb936b5641639d23f375f9f23660d8ff6a34a42ad4"}
Nov 02 10:31:32.100118 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:32.100006    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw" event={"ID":"e5c98f31-5358-46e5-825b-ab24b569f10b","Type":"ContainerDied","Data":"601071eed90aa2afa7e66eae678788b2dc68876b57518d9ab2ce618d18c97139"}
Nov 02 10:31:32.100118 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:32.100028    4449 scope.go:122] "RemoveContainer" containerID="5ef7cb2e4084b0be9cbca2bb936b5641639d23f375f9f23660d8ff6a34a42ad4"
Nov 02 10:31:32.109784 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:32.109764    4449 scope.go:122] "RemoveContainer" containerID="9b651472f6f70fe339a759e86aa53cbf0eafa4727ba38e18089fa52e7a1a6b0f"
Nov 02 10:31:32.117738 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:32.117722    4449 scope.go:122] "RemoveContainer" containerID="5ef7cb2e4084b0be9cbca2bb936b5641639d23f375f9f23660d8ff6a34a42ad4"
Nov 02 10:31:32.117962 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:31:32.117947    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef7cb2e4084b0be9cbca2bb936b5641639d23f375f9f23660d8ff6a34a42ad4\": container with ID starting with 5ef7cb2e4084b0be9cbca2bb936b5641639d23f375f9f23660d8ff6a34a42ad4 not found: ID does not exist" containerID="5ef7cb2e4084b0be9cbca2bb936b5641639d23f375f9f23660d8ff6a34a42ad4"
Nov 02 10:31:32.118006 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:32.117969    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef7cb2e4084b0be9cbca2bb936b5641639d23f375f9f23660d8ff6a34a42ad4"} err="failed to get container status \"5ef7cb2e4084b0be9cbca2bb936b5641639d23f375f9f23660d8ff6a34a42ad4\": rpc error: code = NotFound desc = could not find container \"5ef7cb2e4084b0be9cbca2bb936b5641639d23f375f9f23660d8ff6a34a42ad4\": container with ID starting with 5ef7cb2e4084b0be9cbca2bb936b5641639d23f375f9f23660d8ff6a34a42ad4 not found: ID does not exist"
Nov 02 10:31:32.118006 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:32.117982    4449 scope.go:122] "RemoveContainer" containerID="9b651472f6f70fe339a759e86aa53cbf0eafa4727ba38e18089fa52e7a1a6b0f"
Nov 02 10:31:32.118183 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:31:32.118166    4449 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b651472f6f70fe339a759e86aa53cbf0eafa4727ba38e18089fa52e7a1a6b0f\": container with ID starting with 9b651472f6f70fe339a759e86aa53cbf0eafa4727ba38e18089fa52e7a1a6b0f not found: ID does not exist" containerID="9b651472f6f70fe339a759e86aa53cbf0eafa4727ba38e18089fa52e7a1a6b0f"
Nov 02 10:31:32.118246 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:32.118192    4449 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b651472f6f70fe339a759e86aa53cbf0eafa4727ba38e18089fa52e7a1a6b0f"} err="failed to get container status \"9b651472f6f70fe339a759e86aa53cbf0eafa4727ba38e18089fa52e7a1a6b0f\": rpc error: code = NotFound desc = could not find container \"9b651472f6f70fe339a759e86aa53cbf0eafa4727ba38e18089fa52e7a1a6b0f\": container with ID starting with 9b651472f6f70fe339a759e86aa53cbf0eafa4727ba38e18089fa52e7a1a6b0f not found: ID does not exist"
Nov 02 10:31:32.129619 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:32.129595    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"]
Nov 02 10:31:32.133832 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:32.133812    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-24rkw"]
Nov 02 10:31:33.205343 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:31:33.205314    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" path="/var/lib/kubelet/pods/e5c98f31-5358-46e5-825b-ab24b569f10b/volumes"
Nov 02 10:31:36.544763 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:31:36.544691    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:32:06.544475 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:32:06.544441    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:32:26.981194 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:26.981160    4449 scope.go:122] "RemoveContainer" containerID="709b33a530b08f48e90b3c8eeb9479f16c46488c6e57d1a0ae3f861b59d633a7"
Nov 02 10:32:36.545665 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:32:36.545627    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:32:53.711681 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.711645    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"]
Nov 02 10:32:53.712323 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712304    4449 cpu_manager.go:414] "containerMap: removing container" podUID="3f245b17-d58a-4a81-ab93-b324f817fc28" containerName="cleanup"
Nov 02 10:32:53.712378 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712328    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f245b17-d58a-4a81-ab93-b324f817fc28" containerName="cleanup"
Nov 02 10:32:53.712378 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712350    4449 cpu_manager.go:414] "containerMap: removing container" podUID="3f245b17-d58a-4a81-ab93-b324f817fc28" containerName="cleanup"
Nov 02 10:32:53.712378 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712358    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f245b17-d58a-4a81-ab93-b324f817fc28" containerName="cleanup"
Nov 02 10:32:53.712378 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712372    4449 cpu_manager.go:414] "containerMap: removing container" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy"
Nov 02 10:32:53.712519 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712379    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy"
Nov 02 10:32:53.712519 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712408    4449 cpu_manager.go:414] "containerMap: removing container" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy"
Nov 02 10:32:53.712519 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712416    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy"
Nov 02 10:32:53.712519 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712436    4449 cpu_manager.go:414] "containerMap: removing container" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy"
Nov 02 10:32:53.712519 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712443    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy"
Nov 02 10:32:53.712675 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712564    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f245b17-d58a-4a81-ab93-b324f817fc28" containerName="cleanup"
Nov 02 10:32:53.712675 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712577    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f245b17-d58a-4a81-ab93-b324f817fc28" containerName="cleanup"
Nov 02 10:32:53.712675 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712585    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="98145c71-b388-4093-83be-cf5f18cf1af0" containerName="istio-proxy"
Nov 02 10:32:53.712675 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.712598    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy"
Nov 02 10:32:53.717196 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.717177    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"
Nov 02 10:32:53.724089 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.724038    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:32:53.724553 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.724535    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:32:53.724812 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.724532    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:32:53.725382 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.725360    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:32:53.728734 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.728714    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"]
Nov 02 10:32:53.765841 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.765801    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dbb327c-gk5cq\" (UID: \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"
Nov 02 10:32:53.765841 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.765839    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bnz8\" (UniqueName: \"kubernetes.io/projected/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-kube-api-access-2bnz8\") pod \"maas-api-revoke-sub-3dbb327c-gk5cq\" (UID: \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"
Nov 02 10:32:53.766051 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.765989    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dbb327c-gk5cq\" (UID: \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"
Nov 02 10:32:53.867555 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.867510    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dbb327c-gk5cq\" (UID: \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"
Nov 02 10:32:53.867555 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.867554    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bnz8\" (UniqueName: \"kubernetes.io/projected/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-kube-api-access-2bnz8\") pod \"maas-api-revoke-sub-3dbb327c-gk5cq\" (UID: \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"
Nov 02 10:32:53.867810 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.867645    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dbb327c-gk5cq\" (UID: \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"
Nov 02 10:32:53.868363 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.868340    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dbb327c-gk5cq\" (UID: \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"
Nov 02 10:32:53.869992 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.869969    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dbb327c-gk5cq\" (UID: \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"
Nov 02 10:32:53.880167 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:53.880142    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bnz8\" (UniqueName: \"kubernetes.io/projected/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-kube-api-access-2bnz8\") pod \"maas-api-revoke-sub-3dbb327c-gk5cq\" (UID: \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"
Nov 02 10:32:54.031050 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:54.030970    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"
Nov 02 10:32:54.324419 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:54.324379    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"]
Nov 02 10:32:54.325886 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:32:54.325859    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6ef6dd_6c36_40c5_be18_c72bcae260ac.slice/crio-2550000913d13b822e8bfdb36c3d99681f4c653c15cbd9173018636467c8797c WatchSource:0}: Error finding container 2550000913d13b822e8bfdb36c3d99681f4c653c15cbd9173018636467c8797c: Status 404 returned error can't find the container with id 2550000913d13b822e8bfdb36c3d99681f4c653c15cbd9173018636467c8797c
Nov 02 10:32:54.508035 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:54.507989    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq" event={"ID":"5b6ef6dd-6c36-40c5-be18-c72bcae260ac","Type":"ContainerStarted","Data":"2550000913d13b822e8bfdb36c3d99681f4c653c15cbd9173018636467c8797c"}
Nov 02 10:32:55.517054 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:55.517018    4449 generic.go:356] "Generic (PLEG): container finished" podID="5b6ef6dd-6c36-40c5-be18-c72bcae260ac" containerID="994f354532f98fa7794866c69091c3f365c6f69d4946ea3e8bf0705036a6c14a" exitCode=0
Nov 02 10:32:55.517490 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:55.517104    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq" event={"ID":"5b6ef6dd-6c36-40c5-be18-c72bcae260ac","Type":"ContainerDied","Data":"994f354532f98fa7794866c69091c3f365c6f69d4946ea3e8bf0705036a6c14a"}
Nov 02 10:32:56.662083 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:56.662052    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"
Nov 02 10:32:56.698505 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:56.698463    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-xks-ca\") pod \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\" (UID: \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\") "
Nov 02 10:32:56.698669 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:56.698650    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-kube-api-access-2bnz8\" (UniqueName: \"kubernetes.io/projected/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-kube-api-access-2bnz8\") pod \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\" (UID: \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\") "
Nov 02 10:32:56.698726 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:56.698686    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-openshift-ca\") pod \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\" (UID: \"5b6ef6dd-6c36-40c5-be18-c72bcae260ac\") "
Nov 02 10:32:56.699110 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:56.699079    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-openshift-ca" pod "5b6ef6dd-6c36-40c5-be18-c72bcae260ac" (UID: "5b6ef6dd-6c36-40c5-be18-c72bcae260ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:32:56.700694 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:56.700666    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-xks-ca" pod "5b6ef6dd-6c36-40c5-be18-c72bcae260ac" (UID: "5b6ef6dd-6c36-40c5-be18-c72bcae260ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:32:56.700860 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:56.700834    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-kube-api-access-2bnz8" pod "5b6ef6dd-6c36-40c5-be18-c72bcae260ac" (UID: "5b6ef6dd-6c36-40c5-be18-c72bcae260ac"). InnerVolumeSpecName "kube-api-access-2bnz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:32:56.799541 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:56.799512    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:32:56.799541 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:56.799540    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bnz8\" (UniqueName: \"kubernetes.io/projected/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-kube-api-access-2bnz8\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:32:56.799735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:56.799550    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6ef6dd-6c36-40c5-be18-c72bcae260ac-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:32:57.528724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:57.528646    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"
Nov 02 10:32:57.528875 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:57.528640    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq" event={"ID":"5b6ef6dd-6c36-40c5-be18-c72bcae260ac","Type":"ContainerDied","Data":"2550000913d13b822e8bfdb36c3d99681f4c653c15cbd9173018636467c8797c"}
Nov 02 10:32:57.528875 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:32:57.528763    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2550000913d13b822e8bfdb36c3d99681f4c653c15cbd9173018636467c8797c"
Nov 02 10:33:06.544856 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:33:06.544774    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:33:11.400810 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.400777    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"]
Nov 02 10:33:11.401404 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.401374    4449 cpu_manager.go:414] "containerMap: removing container" podUID="3f245b17-d58a-4a81-ab93-b324f817fc28" containerName="cleanup"
Nov 02 10:33:11.401404 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.401395    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f245b17-d58a-4a81-ab93-b324f817fc28" containerName="cleanup"
Nov 02 10:33:11.401561 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.401464    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5b6ef6dd-6c36-40c5-be18-c72bcae260ac" containerName="revoke-keys"
Nov 02 10:33:11.401561 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.401475    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6ef6dd-6c36-40c5-be18-c72bcae260ac" containerName="revoke-keys"
Nov 02 10:33:11.401724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.401580    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6ef6dd-6c36-40c5-be18-c72bcae260ac" containerName="revoke-keys"
Nov 02 10:33:11.401724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.401593    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f245b17-d58a-4a81-ab93-b324f817fc28" containerName="cleanup"
Nov 02 10:33:11.401724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.401607    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c98f31-5358-46e5-825b-ab24b569f10b" containerName="istio-proxy"
Nov 02 10:33:11.405111 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.405091    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.417737 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.417714    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"]
Nov 02 10:33:11.528921 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.528894    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e601b5-d915-4326-b964-523a4f66788d-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.529064 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.528945    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e601b5-d915-4326-b964-523a4f66788d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.529064 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.528975    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.529193 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.529059    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.529193 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.529101    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e601b5-d915-4326-b964-523a4f66788d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.529357 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.529208    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.529357 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.529282    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.529467 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.529350    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.529467 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.529384    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2tl\" (UniqueName: \"kubernetes.io/projected/51e601b5-d915-4326-b964-523a4f66788d-kube-api-access-xb2tl\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.629908 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.629886    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e601b5-d915-4326-b964-523a4f66788d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.630040 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.629915    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.630040 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.629932    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.630040 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.629950    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e601b5-d915-4326-b964-523a4f66788d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.630040 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.629972    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.630040 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.630025    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.630370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.630059    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.630370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.630085    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2tl\" (UniqueName: \"kubernetes.io/projected/51e601b5-d915-4326-b964-523a4f66788d-kube-api-access-xb2tl\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.630370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.630119    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e601b5-d915-4326-b964-523a4f66788d-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.630539 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.630443    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.630599 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.630551    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.630657 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.630613    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.630871 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.630669    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.630871 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.630814    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e601b5-d915-4326-b964-523a4f66788d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.632316 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.632295    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e601b5-d915-4326-b964-523a4f66788d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.632658 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.632640    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e601b5-d915-4326-b964-523a4f66788d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.642658 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.642638    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e601b5-d915-4326-b964-523a4f66788d-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.643064 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.643041    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2tl\" (UniqueName: \"kubernetes.io/projected/51e601b5-d915-4326-b964-523a4f66788d-kube-api-access-xb2tl\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9nd6j\" (UID: \"51e601b5-d915-4326-b964-523a4f66788d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:11.721104 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:11.721056    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:12.000126 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:12.000096    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"]
Nov 02 10:33:12.005367 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:12.005335    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:33:12.005454 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:12.005400    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:33:12.005454 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:12.005429    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:33:12.604453 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:12.604415    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j" event={"ID":"51e601b5-d915-4326-b964-523a4f66788d","Type":"ContainerStarted","Data":"f2519305d6975900794c97e4c1860482c904f4cac938f4e3e00e272cb13fc398"}
Nov 02 10:33:12.604848 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:12.604473    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j" event={"ID":"51e601b5-d915-4326-b964-523a4f66788d","Type":"ContainerStarted","Data":"96f2d760d4d7ac1b7647d34b5bcac3946f43e621267537e7a6149ec914cc8719"}
Nov 02 10:33:12.631595 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:12.631539    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j" podStartSLOduration=1.631524854 podStartE2EDuration="1.631524854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:33:12.628753417 +0000 UTC m=+3469.567040212" watchObservedRunningTime="2026-11-02 10:33:12.631524854 +0000 UTC m=+3469.569811627"
Nov 02 10:33:12.721663 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:12.721631    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:13.723385 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:13.723349    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 02 10:33:13.723813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:13.723405    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j" podUID="51e601b5-d915-4326-b964-523a4f66788d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 02 10:33:14.722112 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:14.722080    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 02 10:33:14.722310 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:14.722126    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j" podUID="51e601b5-d915-4326-b964-523a4f66788d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 02 10:33:15.721727 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:15.721691    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 02 10:33:15.722109 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:15.721748    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j" podUID="51e601b5-d915-4326-b964-523a4f66788d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 02 10:33:16.721921 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:16.721879    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 02 10:33:16.722426 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:16.721942    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j" podUID="51e601b5-d915-4326-b964-523a4f66788d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 02 10:33:17.721612 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:17.721573    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 02 10:33:17.721806 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:17.721646    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j" podUID="51e601b5-d915-4326-b964-523a4f66788d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 02 10:33:18.722135 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:18.722099    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 02 10:33:18.722543 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:18.722158    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j" podUID="51e601b5-d915-4326-b964-523a4f66788d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 02 10:33:19.721895 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:19.721858    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 02 10:33:19.722105 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:19.721924    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j" podUID="51e601b5-d915-4326-b964-523a4f66788d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 02 10:33:20.970671 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:20.970637    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:20.971185 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:20.971164    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:20.971893 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:20.971872    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j"
Nov 02 10:33:26.425440 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.425405    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"]
Nov 02 10:33:26.429467 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.429444    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.441371 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.441348    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"]
Nov 02 10:33:26.483277 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.483243    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.483404 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.483282    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.483404 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.483311    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.483491 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.483426    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.483491 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.483481    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.483565 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.483513    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2l8r\" (UniqueName: \"kubernetes.io/projected/604d7b4e-efdd-4979-b544-fd3d4d1953a3-kube-api-access-w2l8r\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.483565 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.483553    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.483633 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.483594    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.483633 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.483616    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.584258 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.584195    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.584423 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.584273    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.584423 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.584294    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.584423 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.584331    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.584423 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.584355    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.584423 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.584400    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.584423 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.584424    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.584764 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.584450    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.584764 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.584496    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2l8r\" (UniqueName: \"kubernetes.io/projected/604d7b4e-efdd-4979-b544-fd3d4d1953a3-kube-api-access-w2l8r\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.584764 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.584622    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.584931 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.584761    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.584931 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.584808    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.585060 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.585037    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.585184 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.585167    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.586803 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.586778    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.586935 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.586918    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.596998 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.596973    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604d7b4e-efdd-4979-b544-fd3d4d1953a3-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.597392 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.597371    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2l8r\" (UniqueName: \"kubernetes.io/projected/604d7b4e-efdd-4979-b544-fd3d4d1953a3-kube-api-access-w2l8r\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-6cftx\" (UID: \"604d7b4e-efdd-4979-b544-fd3d4d1953a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:26.747256 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:26.747172    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:27.026441 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:27.026419    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"]
Nov 02 10:33:27.027349 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:33:27.027324    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604d7b4e_efdd_4979_b544_fd3d4d1953a3.slice/crio-ef341d9680838fcaf1e79ebdb6e1b5e081af6754b9ffdcd1bbbe2bc446c06ced WatchSource:0}: Error finding container ef341d9680838fcaf1e79ebdb6e1b5e081af6754b9ffdcd1bbbe2bc446c06ced: Status 404 returned error can't find the container with id ef341d9680838fcaf1e79ebdb6e1b5e081af6754b9ffdcd1bbbe2bc446c06ced
Nov 02 10:33:27.029497 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:27.029464    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:33:27.029623 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:27.029544    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:33:27.029623 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:27.029590    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:33:27.686184 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:27.686145    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx" event={"ID":"604d7b4e-efdd-4979-b544-fd3d4d1953a3","Type":"ContainerStarted","Data":"4cd2db6356b237611408f0f8add239eb889fb6162b2af41bfcea75d4087ba9d5"}
Nov 02 10:33:27.686669 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:27.686211    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx" event={"ID":"604d7b4e-efdd-4979-b544-fd3d4d1953a3","Type":"ContainerStarted","Data":"ef341d9680838fcaf1e79ebdb6e1b5e081af6754b9ffdcd1bbbe2bc446c06ced"}
Nov 02 10:33:27.718288 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:27.718244    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx" podStartSLOduration=1.718232043 podStartE2EDuration="1.718232043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:33:27.716283467 +0000 UTC m=+3484.654570242" watchObservedRunningTime="2026-11-02 10:33:27.718232043 +0000 UTC m=+3484.656518817"
Nov 02 10:33:27.747463 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:27.747439    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:28.756248 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:28.749275    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6cftx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body=
Nov 02 10:33:28.764243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:28.756813    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx" podUID="604d7b4e-efdd-4979-b544-fd3d4d1953a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused"
Nov 02 10:33:29.748072 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:29.748044    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6cftx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body=
Nov 02 10:33:29.748291 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:29.748091    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx" podUID="604d7b4e-efdd-4979-b544-fd3d4d1953a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused"
Nov 02 10:33:30.748328 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:30.748299    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6cftx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body=
Nov 02 10:33:30.748726 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:30.748347    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx" podUID="604d7b4e-efdd-4979-b544-fd3d4d1953a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused"
Nov 02 10:33:31.748506 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:31.748472    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6cftx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body=
Nov 02 10:33:31.748888 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:31.748525    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx" podUID="604d7b4e-efdd-4979-b544-fd3d4d1953a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused"
Nov 02 10:33:32.747887 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:32.747853    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6cftx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body=
Nov 02 10:33:32.748068 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:32.747906    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx" podUID="604d7b4e-efdd-4979-b544-fd3d4d1953a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused"
Nov 02 10:33:33.748471 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:33.748434    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6cftx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body=
Nov 02 10:33:33.748881 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:33.748500    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx" podUID="604d7b4e-efdd-4979-b544-fd3d4d1953a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused"
Nov 02 10:33:34.748463 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:34.748431    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6cftx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body=
Nov 02 10:33:34.748622 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:34.748476    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx" podUID="604d7b4e-efdd-4979-b544-fd3d4d1953a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused"
Nov 02 10:33:36.006971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:36.006932    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:36.007453 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:36.007363    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:36.008015 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:33:36.007996    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx"
Nov 02 10:33:36.545531 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:33:36.545496    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:34:06.544571 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:34:06.544538    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:34:34.603825 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.603749    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"]
Nov 02 10:34:34.607394 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.607377    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"
Nov 02 10:34:34.612993 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.612326    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:34:34.612993 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.612366    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:34:34.613529 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.613508    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:34:34.613657 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.613553    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:34:34.616680 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.616656    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"]
Nov 02 10:34:34.651053 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.651022    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dm7h\" (UniqueName: \"kubernetes.io/projected/5f476720-58b9-4bae-993f-b3f300ea65e9-kube-api-access-5dm7h\") pod \"maas-api-revoke-sub-80d7a79e-g6b7q\" (UID: \"5f476720-58b9-4bae-993f-b3f300ea65e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"
Nov 02 10:34:34.651271 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.651140    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80d7a79e-g6b7q\" (UID: \"5f476720-58b9-4bae-993f-b3f300ea65e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"
Nov 02 10:34:34.651271 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.651196    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80d7a79e-g6b7q\" (UID: \"5f476720-58b9-4bae-993f-b3f300ea65e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"
Nov 02 10:34:34.752124 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.752097    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80d7a79e-g6b7q\" (UID: \"5f476720-58b9-4bae-993f-b3f300ea65e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"
Nov 02 10:34:34.752265 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.752150    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80d7a79e-g6b7q\" (UID: \"5f476720-58b9-4bae-993f-b3f300ea65e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"
Nov 02 10:34:34.752265 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.752201    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dm7h\" (UniqueName: \"kubernetes.io/projected/5f476720-58b9-4bae-993f-b3f300ea65e9-kube-api-access-5dm7h\") pod \"maas-api-revoke-sub-80d7a79e-g6b7q\" (UID: \"5f476720-58b9-4bae-993f-b3f300ea65e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"
Nov 02 10:34:34.752869 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.752845    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80d7a79e-g6b7q\" (UID: \"5f476720-58b9-4bae-993f-b3f300ea65e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"
Nov 02 10:34:34.754520 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.754500    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80d7a79e-g6b7q\" (UID: \"5f476720-58b9-4bae-993f-b3f300ea65e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"
Nov 02 10:34:34.764999 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.764978    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dm7h\" (UniqueName: \"kubernetes.io/projected/5f476720-58b9-4bae-993f-b3f300ea65e9-kube-api-access-5dm7h\") pod \"maas-api-revoke-sub-80d7a79e-g6b7q\" (UID: \"5f476720-58b9-4bae-993f-b3f300ea65e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"
Nov 02 10:34:34.922971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:34.922925    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"
Nov 02 10:34:35.201169 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:34:35.201143    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f476720_58b9_4bae_993f_b3f300ea65e9.slice/crio-1490b116e1bbf50bbce21c911cdea853069c1ae3f4de38bdfe10573c9d30a360 WatchSource:0}: Error finding container 1490b116e1bbf50bbce21c911cdea853069c1ae3f4de38bdfe10573c9d30a360: Status 404 returned error can't find the container with id 1490b116e1bbf50bbce21c911cdea853069c1ae3f4de38bdfe10573c9d30a360
Nov 02 10:34:35.207582 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:35.207557    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"]
Nov 02 10:34:36.032831 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:36.032800    4449 generic.go:356] "Generic (PLEG): container finished" podID="5f476720-58b9-4bae-993f-b3f300ea65e9" containerID="9a370065c505ffc8c1a1a87a9b48d4c0642dc1a78f28abfb7c0da34b64ff03e4" exitCode=0
Nov 02 10:34:36.033157 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:36.032881    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q" event={"ID":"5f476720-58b9-4bae-993f-b3f300ea65e9","Type":"ContainerDied","Data":"9a370065c505ffc8c1a1a87a9b48d4c0642dc1a78f28abfb7c0da34b64ff03e4"}
Nov 02 10:34:36.033157 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:36.032933    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q" event={"ID":"5f476720-58b9-4bae-993f-b3f300ea65e9","Type":"ContainerStarted","Data":"1490b116e1bbf50bbce21c911cdea853069c1ae3f4de38bdfe10573c9d30a360"}
Nov 02 10:34:36.544576 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:34:36.544538    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:34:37.172745 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:37.172711    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"
Nov 02 10:34:37.276626 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:37.276600    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-xks-ca\") pod \"5f476720-58b9-4bae-993f-b3f300ea65e9\" (UID: \"5f476720-58b9-4bae-993f-b3f300ea65e9\") "
Nov 02 10:34:37.276774 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:37.276666    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f476720-58b9-4bae-993f-b3f300ea65e9-kube-api-access-5dm7h\" (UniqueName: \"kubernetes.io/projected/5f476720-58b9-4bae-993f-b3f300ea65e9-kube-api-access-5dm7h\") pod \"5f476720-58b9-4bae-993f-b3f300ea65e9\" (UID: \"5f476720-58b9-4bae-993f-b3f300ea65e9\") "
Nov 02 10:34:37.276774 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:37.276708    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-openshift-ca\") pod \"5f476720-58b9-4bae-993f-b3f300ea65e9\" (UID: \"5f476720-58b9-4bae-993f-b3f300ea65e9\") "
Nov 02 10:34:37.277122 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:37.277090    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-openshift-ca" pod "5f476720-58b9-4bae-993f-b3f300ea65e9" (UID: "5f476720-58b9-4bae-993f-b3f300ea65e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:34:37.278748 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:37.278725    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-xks-ca" pod "5f476720-58b9-4bae-993f-b3f300ea65e9" (UID: "5f476720-58b9-4bae-993f-b3f300ea65e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:34:37.278848 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:37.278832    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f476720-58b9-4bae-993f-b3f300ea65e9-kube-api-access-5dm7h" pod "5f476720-58b9-4bae-993f-b3f300ea65e9" (UID: "5f476720-58b9-4bae-993f-b3f300ea65e9"). InnerVolumeSpecName "kube-api-access-5dm7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:34:37.377929 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:37.377875    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:34:37.377929 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:37.377897    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dm7h\" (UniqueName: \"kubernetes.io/projected/5f476720-58b9-4bae-993f-b3f300ea65e9-kube-api-access-5dm7h\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:34:37.377929 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:37.377906    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f476720-58b9-4bae-993f-b3f300ea65e9-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:34:38.046180 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:38.046153    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"
Nov 02 10:34:38.046346 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:38.046186    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q" event={"ID":"5f476720-58b9-4bae-993f-b3f300ea65e9","Type":"ContainerDied","Data":"1490b116e1bbf50bbce21c911cdea853069c1ae3f4de38bdfe10573c9d30a360"}
Nov 02 10:34:38.046346 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:34:38.046264    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1490b116e1bbf50bbce21c911cdea853069c1ae3f4de38bdfe10573c9d30a360"
Nov 02 10:35:06.544166 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:35:06.544132    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:35:11.564129 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.564100    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"]
Nov 02 10:35:11.564718 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.564698    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5f476720-58b9-4bae-993f-b3f300ea65e9" containerName="revoke-keys"
Nov 02 10:35:11.564807 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.564720    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f476720-58b9-4bae-993f-b3f300ea65e9" containerName="revoke-keys"
Nov 02 10:35:11.564886 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.564873    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f476720-58b9-4bae-993f-b3f300ea65e9" containerName="revoke-keys"
Nov 02 10:35:11.568245 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.568210    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.582929 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.582882    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"]
Nov 02 10:35:11.681115 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.681083    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.681273 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.681123    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de093953-9e5f-4d01-9e8f-0404f29e0b82-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.681273 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.681204    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de093953-9e5f-4d01-9e8f-0404f29e0b82-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.681273 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.681261    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.681457 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.681283    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpv4f\" (UniqueName: \"kubernetes.io/projected/de093953-9e5f-4d01-9e8f-0404f29e0b82-kube-api-access-gpv4f\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.681457 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.681414    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de093953-9e5f-4d01-9e8f-0404f29e0b82-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.681574 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.681459    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.681611 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.681587    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.681645 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.681612    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.781960 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.781934    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.782089 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.781966    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.782089 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.781991    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.782089 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.782008    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de093953-9e5f-4d01-9e8f-0404f29e0b82-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.782089 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.782044    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de093953-9e5f-4d01-9e8f-0404f29e0b82-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.782089 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.782069    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.782420 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.782094    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpv4f\" (UniqueName: \"kubernetes.io/projected/de093953-9e5f-4d01-9e8f-0404f29e0b82-kube-api-access-gpv4f\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.782420 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.782135    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de093953-9e5f-4d01-9e8f-0404f29e0b82-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.782420 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.782171    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.782420 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.782390    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.782568 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.782430    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.782658 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.782634    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.782707 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.782653    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.782944 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.782926    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de093953-9e5f-4d01-9e8f-0404f29e0b82-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.784267 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.784238    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de093953-9e5f-4d01-9e8f-0404f29e0b82-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.784533 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.784517    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de093953-9e5f-4d01-9e8f-0404f29e0b82-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.796102 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.796079    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de093953-9e5f-4d01-9e8f-0404f29e0b82-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.796336 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.796318    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpv4f\" (UniqueName: \"kubernetes.io/projected/de093953-9e5f-4d01-9e8f-0404f29e0b82-kube-api-access-gpv4f\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-mqzmk\" (UID: \"de093953-9e5f-4d01-9e8f-0404f29e0b82\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:11.893735 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:11.893686    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:12.173771 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:12.173743    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"]
Nov 02 10:35:12.174745 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:35:12.174718    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde093953_9e5f_4d01_9e8f_0404f29e0b82.slice/crio-e228cbd36f184bef128883e6a8067d9bccfc0f4a560b820742e82edcc6566c36 WatchSource:0}: Error finding container e228cbd36f184bef128883e6a8067d9bccfc0f4a560b820742e82edcc6566c36: Status 404 returned error can't find the container with id e228cbd36f184bef128883e6a8067d9bccfc0f4a560b820742e82edcc6566c36
Nov 02 10:35:12.176837 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:12.176808    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:35:12.176898 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:12.176870    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:35:12.176936 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:12.176900    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:35:12.231249 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:12.231203    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk" event={"ID":"de093953-9e5f-4d01-9e8f-0404f29e0b82","Type":"ContainerStarted","Data":"e228cbd36f184bef128883e6a8067d9bccfc0f4a560b820742e82edcc6566c36"}
Nov 02 10:35:13.237427 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:13.237389    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk" event={"ID":"de093953-9e5f-4d01-9e8f-0404f29e0b82","Type":"ContainerStarted","Data":"3b77eb81d6a8e3660969268c656f2314f0b9273277b3f941cf9abb009fa57557"}
Nov 02 10:35:13.278199 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:13.278145    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk" podStartSLOduration=2.278131437 podStartE2EDuration="2.278131437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:35:13.274852017 +0000 UTC m=+3590.213138858" watchObservedRunningTime="2026-11-02 10:35:13.278131437 +0000 UTC m=+3590.216418210"
Nov 02 10:35:13.894751 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:13.894717    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:13.896056 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:13.896032    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body=
Nov 02 10:35:13.896192 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:13.896076    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk" podUID="de093953-9e5f-4d01-9e8f-0404f29e0b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused"
Nov 02 10:35:14.894404 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:14.894371    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body=
Nov 02 10:35:14.894801 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:14.894428    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk" podUID="de093953-9e5f-4d01-9e8f-0404f29e0b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused"
Nov 02 10:35:15.894976 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:15.894943    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body=
Nov 02 10:35:15.895405 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:15.894993    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk" podUID="de093953-9e5f-4d01-9e8f-0404f29e0b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused"
Nov 02 10:35:16.894759 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:16.894723    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body=
Nov 02 10:35:16.894942 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:16.894779    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk" podUID="de093953-9e5f-4d01-9e8f-0404f29e0b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused"
Nov 02 10:35:17.894967 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:17.894934    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body=
Nov 02 10:35:17.895391 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:17.894989    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk" podUID="de093953-9e5f-4d01-9e8f-0404f29e0b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused"
Nov 02 10:35:18.894497 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:18.894467    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body=
Nov 02 10:35:18.894680 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:18.894521    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk" podUID="de093953-9e5f-4d01-9e8f-0404f29e0b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused"
Nov 02 10:35:20.872894 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:20.872860    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:20.873477 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:20.873456    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:20.874130 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:35:20.874112    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk"
Nov 02 10:35:36.544654 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:35:36.544618    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:36:06.545539 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:36:06.545454    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:36:15.964272 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:15.964232    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"]
Nov 02 10:36:15.968649 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:15.968625    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:15.972330 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:15.972303    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ce5e7da4-openshift-default-dockercfg-rclm5\""
Nov 02 10:36:15.983265 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:15.983240    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"]
Nov 02 10:36:16.112105 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.112074    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-certs\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.112283 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.112116    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkxtr\" (UniqueName: \"kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-kube-api-access-kkxtr\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.112283 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.112142    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-credential-socket\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.112725 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.112287    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89113bc-dec8-4aea-9aca-79a234d9b25b-istiod-ca-cert\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.112725 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.112332    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-data\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.112725 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.112410    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-socket\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.112725 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.112505    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-envoy\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.112725 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.112530    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-podinfo\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.112725 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.112550    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-token\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.214004 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.213954    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-envoy\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.214004 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.214006    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-podinfo\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.214358 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.214024    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-token\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.214358 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.214054    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-certs\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.214358 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.214089    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkxtr\" (UniqueName: \"kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-kube-api-access-kkxtr\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.214358 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.214112    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-credential-socket\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.214358 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.214139    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89113bc-dec8-4aea-9aca-79a234d9b25b-istiod-ca-cert\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.214358 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.214185    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-data\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.214358 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.214299    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-socket\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.214771 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.214737    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-certs\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.214949 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.214924    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-socket\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.215080 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.215062    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89113bc-dec8-4aea-9aca-79a234d9b25b-istiod-ca-cert\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.215138 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.215086    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-credential-socket\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.215298 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.215283    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-data\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.216411 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.216390    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-envoy\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.216665 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.216648    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-podinfo\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.229137 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.229112    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-token\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.231740 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.231718    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkxtr\" (UniqueName: \"kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-kube-api-access-kkxtr\") pod \"e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") " pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.287107 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.287083    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:16.568407 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.568377    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"]
Nov 02 10:36:16.569701 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:36:16.569679    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89113bc_dec8_4aea_9aca_79a234d9b25b.slice/crio-0167d30f3fd45c57b680042b3ab2d6796ef882cf7991fdb5f224a2273ffb175d WatchSource:0}: Error finding container 0167d30f3fd45c57b680042b3ab2d6796ef882cf7991fdb5f224a2273ffb175d: Status 404 returned error can't find the container with id 0167d30f3fd45c57b680042b3ab2d6796ef882cf7991fdb5f224a2273ffb175d
Nov 02 10:36:16.572004 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.571972    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:36:16.572117 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.572042    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:36:16.572117 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.572072    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:36:16.581240 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:16.579189    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" event={"ID":"c89113bc-dec8-4aea-9aca-79a234d9b25b","Type":"ContainerStarted","Data":"0167d30f3fd45c57b680042b3ab2d6796ef882cf7991fdb5f224a2273ffb175d"}
Nov 02 10:36:17.585835 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:17.585798    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" event={"ID":"c89113bc-dec8-4aea-9aca-79a234d9b25b","Type":"ContainerStarted","Data":"09338d153ddb9a27a416a1643e042374a711bf5db3eae8b357dd24c3e2c58a80"}
Nov 02 10:36:17.611419 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:17.611369    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" podStartSLOduration=2.611357462 podStartE2EDuration="2.611357462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:36:17.608406394 +0000 UTC m=+3654.546693189" watchObservedRunningTime="2026-11-02 10:36:17.611357462 +0000 UTC m=+3654.549644235"
Nov 02 10:36:18.288089 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:18.288061    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:18.292446 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:18.292418    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:18.591587 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:18.591519    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:18.592258 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:18.592241    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:32.722561 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:32.722473    4449 generic.go:356] "Generic (PLEG): container finished" podID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerID="09338d153ddb9a27a416a1643e042374a711bf5db3eae8b357dd24c3e2c58a80" exitCode=0
Nov 02 10:36:32.722561 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:32.722548    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" event={"ID":"c89113bc-dec8-4aea-9aca-79a234d9b25b","Type":"ContainerDied","Data":"09338d153ddb9a27a416a1643e042374a711bf5db3eae8b357dd24c3e2c58a80"}
Nov 02 10:36:32.722985 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:32.722886    4449 scope.go:122] "RemoveContainer" containerID="09338d153ddb9a27a416a1643e042374a711bf5db3eae8b357dd24c3e2c58a80"
Nov 02 10:36:32.724024 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:32.723999    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:36:32.724077 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:32.724063    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:36:32.724112 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:32.724093    4449 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 10:36:33.591779 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:33.591726    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:33.729527 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:33.729487    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" event={"ID":"c89113bc-dec8-4aea-9aca-79a234d9b25b","Type":"ContainerStarted","Data":"3918fd5fd1172778b9a5fad835321cfe9f1da795715680514686519dbce6fbde"}
Nov 02 10:36:34.287564 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:34.287521    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:34.288807 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:34.288783    4449 patch_prober.go:29] interesting pod/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body=
Nov 02 10:36:34.288962 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:34.288827    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused"
Nov 02 10:36:35.287921 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:35.287889    4449 patch_prober.go:29] interesting pod/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body=
Nov 02 10:36:35.288350 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:35.287943    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused"
Nov 02 10:36:36.287784 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:36.287741    4449 patch_prober.go:29] interesting pod/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body=
Nov 02 10:36:36.288037 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:36.287813    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused"
Nov 02 10:36:36.545967 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:36:36.545890    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:36:37.287523 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:37.287492    4449 patch_prober.go:29] interesting pod/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body=
Nov 02 10:36:37.287703 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:37.287540    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused"
Nov 02 10:36:38.288049 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:38.288015    4449 patch_prober.go:29] interesting pod/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body=
Nov 02 10:36:38.288447 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:38.288071    4449 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused"
Nov 02 10:36:39.292203 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:39.292168    4449 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:39.292722 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:39.292521    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:36:39.293275 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:36:39.293259    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:37:06.544515 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:37:06.544479    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:37:36.544997 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:37:36.544925    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:37:57.043869 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:37:57.043825    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"]
Nov 02 10:37:57.046661 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:37:57.046635    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dbb327c-gk5cq"]
Nov 02 10:37:57.205131 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:37:57.205088    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6ef6dd-6c36-40c5-be18-c72bcae260ac" path="/var/lib/kubelet/pods/5b6ef6dd-6c36-40c5-be18-c72bcae260ac/volumes"
Nov 02 10:38:06.545277 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:38:06.545240    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:38:20.837419 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:20.837389    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"]
Nov 02 10:38:20.837896 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:20.837643    4449 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerName="istio-proxy" containerID="cri-o://3918fd5fd1172778b9a5fad835321cfe9f1da795715680514686519dbce6fbde" gracePeriod=30
Nov 02 10:38:24.295094 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:24.295047    4449 patch_prober.go:29] interesting pod/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 10:38:24.295638 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:24.295134    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 10:38:26.590300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.573248    4449 generic.go:356] "Generic (PLEG): container finished" podID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerID="3918fd5fd1172778b9a5fad835321cfe9f1da795715680514686519dbce6fbde" exitCode=0
Nov 02 10:38:26.590300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.573401    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" event={"ID":"c89113bc-dec8-4aea-9aca-79a234d9b25b","Type":"ContainerDied","Data":"3918fd5fd1172778b9a5fad835321cfe9f1da795715680514686519dbce6fbde"}
Nov 02 10:38:26.590300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.573457    4449 scope.go:122] "RemoveContainer" containerID="09338d153ddb9a27a416a1643e042374a711bf5db3eae8b357dd24c3e2c58a80"
Nov 02 10:38:26.721732 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.721390    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:38:26.877473 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.845685    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-envoy\") pod \"c89113bc-dec8-4aea-9aca-79a234d9b25b\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") "
Nov 02 10:38:26.877473 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.845749    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-data\") pod \"c89113bc-dec8-4aea-9aca-79a234d9b25b\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") "
Nov 02 10:38:26.877473 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.845813    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-token\" (UniqueName: \"kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-token\") pod \"c89113bc-dec8-4aea-9aca-79a234d9b25b\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") "
Nov 02 10:38:26.877473 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.845848    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-credential-socket\") pod \"c89113bc-dec8-4aea-9aca-79a234d9b25b\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") "
Nov 02 10:38:26.877473 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.845877    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-kube-api-access-kkxtr\" (UniqueName: \"kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-kube-api-access-kkxtr\") pod \"c89113bc-dec8-4aea-9aca-79a234d9b25b\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") "
Nov 02 10:38:26.877473 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.845923    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-socket\") pod \"c89113bc-dec8-4aea-9aca-79a234d9b25b\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") "
Nov 02 10:38:26.877473 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.845984    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89113bc-dec8-4aea-9aca-79a234d9b25b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89113bc-dec8-4aea-9aca-79a234d9b25b-istiod-ca-cert\") pod \"c89113bc-dec8-4aea-9aca-79a234d9b25b\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") "
Nov 02 10:38:26.877473 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.846036    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-podinfo\") pod \"c89113bc-dec8-4aea-9aca-79a234d9b25b\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") "
Nov 02 10:38:26.877473 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.846096    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-certs\") pod \"c89113bc-dec8-4aea-9aca-79a234d9b25b\" (UID: \"c89113bc-dec8-4aea-9aca-79a234d9b25b\") "
Nov 02 10:38:26.923244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.918443    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-socket" pod "c89113bc-dec8-4aea-9aca-79a234d9b25b" (UID: "c89113bc-dec8-4aea-9aca-79a234d9b25b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:38:26.923244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.918754    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89113bc-dec8-4aea-9aca-79a234d9b25b-istiod-ca-cert" pod "c89113bc-dec8-4aea-9aca-79a234d9b25b" (UID: "c89113bc-dec8-4aea-9aca-79a234d9b25b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:38:26.932498 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.932448    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-certs" pod "c89113bc-dec8-4aea-9aca-79a234d9b25b" (UID: "c89113bc-dec8-4aea-9aca-79a234d9b25b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:38:26.965285 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.962749    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-data" pod "c89113bc-dec8-4aea-9aca-79a234d9b25b" (UID: "c89113bc-dec8-4aea-9aca-79a234d9b25b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:38:26.965285 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.963004    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-credential-socket" pod "c89113bc-dec8-4aea-9aca-79a234d9b25b" (UID: "c89113bc-dec8-4aea-9aca-79a234d9b25b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:38:26.965285 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.964793    4449 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-certs\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:26.965285 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.964819    4449 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-data\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:26.965285 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.964856    4449 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-credential-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:26.965285 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.964873    4449 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-workload-socket\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:26.965285 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:26.964887    4449 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89113bc-dec8-4aea-9aca-79a234d9b25b-istiod-ca-cert\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:27.049889 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:27.049836    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-token" pod "c89113bc-dec8-4aea-9aca-79a234d9b25b" (UID: "c89113bc-dec8-4aea-9aca-79a234d9b25b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:38:27.080439 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:27.052314    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-podinfo" pod "c89113bc-dec8-4aea-9aca-79a234d9b25b" (UID: "c89113bc-dec8-4aea-9aca-79a234d9b25b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:38:27.080439 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:27.078303    4449 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-token\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:27.080439 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:27.078330    4449 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-podinfo\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:27.080439 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:27.079386    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-kube-api-access-kkxtr" pod "c89113bc-dec8-4aea-9aca-79a234d9b25b" (UID: "c89113bc-dec8-4aea-9aca-79a234d9b25b"). InnerVolumeSpecName "kube-api-access-kkxtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:38:27.080439 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:27.079467    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-envoy" pod "c89113bc-dec8-4aea-9aca-79a234d9b25b" (UID: "c89113bc-dec8-4aea-9aca-79a234d9b25b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:38:27.190871 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:27.190773    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkxtr\" (UniqueName: \"kubernetes.io/projected/c89113bc-dec8-4aea-9aca-79a234d9b25b-kube-api-access-kkxtr\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:27.191099 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:27.191070    4449 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89113bc-dec8-4aea-9aca-79a234d9b25b-istio-envoy\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:27.616103 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:27.616043    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf" event={"ID":"c89113bc-dec8-4aea-9aca-79a234d9b25b","Type":"ContainerDied","Data":"0167d30f3fd45c57b680042b3ab2d6796ef882cf7991fdb5f224a2273ffb175d"}
Nov 02 10:38:27.616670 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:27.616342    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"
Nov 02 10:38:27.616670 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:27.616571    4449 scope.go:122] "RemoveContainer" containerID="3918fd5fd1172778b9a5fad835321cfe9f1da795715680514686519dbce6fbde"
Nov 02 10:38:27.743315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:27.743277    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"]
Nov 02 10:38:27.750244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:27.749726    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ce5e7da4-openshift-default-5fb74bb54f-czsxf"]
Nov 02 10:38:29.209653 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:29.209602    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" path="/var/lib/kubelet/pods/c89113bc-dec8-4aea-9aca-79a234d9b25b/volumes"
Nov 02 10:38:33.268667 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.268636    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"]
Nov 02 10:38:33.269133 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.269118    4449 cpu_manager.go:414] "containerMap: removing container" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerName="istio-proxy"
Nov 02 10:38:33.269201 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.269134    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerName="istio-proxy"
Nov 02 10:38:33.269201 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.269146    4449 cpu_manager.go:414] "containerMap: removing container" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerName="istio-proxy"
Nov 02 10:38:33.269201 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.269151    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerName="istio-proxy"
Nov 02 10:38:33.269323 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.269231    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerName="istio-proxy"
Nov 02 10:38:33.269323 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.269240    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89113bc-dec8-4aea-9aca-79a234d9b25b" containerName="istio-proxy"
Nov 02 10:38:33.272401 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.272387    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"
Nov 02 10:38:33.278439 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.278413    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:38:33.279346 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.279328    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:38:33.279447 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.279412    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:38:33.279509 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.279441    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:38:33.283356 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.283330    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"]
Nov 02 10:38:33.345347 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.345323    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a0e87e7-wfzr2\" (UID: \"e95f40d2-3c86-4c43-8c40-b02c90413a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"
Nov 02 10:38:33.345470 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.345403    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a0e87e7-wfzr2\" (UID: \"e95f40d2-3c86-4c43-8c40-b02c90413a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"
Nov 02 10:38:33.345551 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.345517    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trnxd\" (UniqueName: \"kubernetes.io/projected/e95f40d2-3c86-4c43-8c40-b02c90413a19-kube-api-access-trnxd\") pod \"maas-api-revoke-sub-7a0e87e7-wfzr2\" (UID: \"e95f40d2-3c86-4c43-8c40-b02c90413a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"
Nov 02 10:38:33.446475 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.446456    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnxd\" (UniqueName: \"kubernetes.io/projected/e95f40d2-3c86-4c43-8c40-b02c90413a19-kube-api-access-trnxd\") pod \"maas-api-revoke-sub-7a0e87e7-wfzr2\" (UID: \"e95f40d2-3c86-4c43-8c40-b02c90413a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"
Nov 02 10:38:33.446571 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.446500    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a0e87e7-wfzr2\" (UID: \"e95f40d2-3c86-4c43-8c40-b02c90413a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"
Nov 02 10:38:33.446571 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.446544    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a0e87e7-wfzr2\" (UID: \"e95f40d2-3c86-4c43-8c40-b02c90413a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"
Nov 02 10:38:33.447236 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.447188    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a0e87e7-wfzr2\" (UID: \"e95f40d2-3c86-4c43-8c40-b02c90413a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"
Nov 02 10:38:33.449109 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.449087    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a0e87e7-wfzr2\" (UID: \"e95f40d2-3c86-4c43-8c40-b02c90413a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"
Nov 02 10:38:33.458744 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.458726    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trnxd\" (UniqueName: \"kubernetes.io/projected/e95f40d2-3c86-4c43-8c40-b02c90413a19-kube-api-access-trnxd\") pod \"maas-api-revoke-sub-7a0e87e7-wfzr2\" (UID: \"e95f40d2-3c86-4c43-8c40-b02c90413a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"
Nov 02 10:38:33.586475 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.586458    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"
Nov 02 10:38:33.860905 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:33.860877    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"]
Nov 02 10:38:33.861963 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:38:33.861936    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95f40d2_3c86_4c43_8c40_b02c90413a19.slice/crio-13039c2dba5cafdc54fa1c1db97b3fe7f2aed96d4770979b5a3f4c3122c575a1 WatchSource:0}: Error finding container 13039c2dba5cafdc54fa1c1db97b3fe7f2aed96d4770979b5a3f4c3122c575a1: Status 404 returned error can't find the container with id 13039c2dba5cafdc54fa1c1db97b3fe7f2aed96d4770979b5a3f4c3122c575a1
Nov 02 10:38:34.711150 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:34.711126    4449 generic.go:356] "Generic (PLEG): container finished" podID="e95f40d2-3c86-4c43-8c40-b02c90413a19" containerID="89e192a731e82bf2128816d1887988d488700cf9ca90a4ca51cce3d9469fe8a4" exitCode=0
Nov 02 10:38:34.711466 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:34.711193    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2" event={"ID":"e95f40d2-3c86-4c43-8c40-b02c90413a19","Type":"ContainerDied","Data":"89e192a731e82bf2128816d1887988d488700cf9ca90a4ca51cce3d9469fe8a4"}
Nov 02 10:38:34.711466 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:34.711247    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2" event={"ID":"e95f40d2-3c86-4c43-8c40-b02c90413a19","Type":"ContainerStarted","Data":"13039c2dba5cafdc54fa1c1db97b3fe7f2aed96d4770979b5a3f4c3122c575a1"}
Nov 02 10:38:35.856941 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:35.856909    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"
Nov 02 10:38:35.972781 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:35.972711    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-openshift-ca\") pod \"e95f40d2-3c86-4c43-8c40-b02c90413a19\" (UID: \"e95f40d2-3c86-4c43-8c40-b02c90413a19\") "
Nov 02 10:38:35.972781 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:35.972743    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-xks-ca\") pod \"e95f40d2-3c86-4c43-8c40-b02c90413a19\" (UID: \"e95f40d2-3c86-4c43-8c40-b02c90413a19\") "
Nov 02 10:38:35.972952 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:35.972840    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95f40d2-3c86-4c43-8c40-b02c90413a19-kube-api-access-trnxd\" (UniqueName: \"kubernetes.io/projected/e95f40d2-3c86-4c43-8c40-b02c90413a19-kube-api-access-trnxd\") pod \"e95f40d2-3c86-4c43-8c40-b02c90413a19\" (UID: \"e95f40d2-3c86-4c43-8c40-b02c90413a19\") "
Nov 02 10:38:35.973107 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:35.973079    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-openshift-ca" pod "e95f40d2-3c86-4c43-8c40-b02c90413a19" (UID: "e95f40d2-3c86-4c43-8c40-b02c90413a19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:38:35.974851 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:35.974829    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-xks-ca" pod "e95f40d2-3c86-4c43-8c40-b02c90413a19" (UID: "e95f40d2-3c86-4c43-8c40-b02c90413a19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:38:35.974968 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:35.974894    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95f40d2-3c86-4c43-8c40-b02c90413a19-kube-api-access-trnxd" pod "e95f40d2-3c86-4c43-8c40-b02c90413a19" (UID: "e95f40d2-3c86-4c43-8c40-b02c90413a19"). InnerVolumeSpecName "kube-api-access-trnxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:38:36.073900 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:36.073874    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:36.073900 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:36.073898    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95f40d2-3c86-4c43-8c40-b02c90413a19-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:36.074039 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:36.073908    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trnxd\" (UniqueName: \"kubernetes.io/projected/e95f40d2-3c86-4c43-8c40-b02c90413a19-kube-api-access-trnxd\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:36.545066 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:38:36.545037    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:38:36.722340 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:36.722316    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"
Nov 02 10:38:36.722464 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:36.722347    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2" event={"ID":"e95f40d2-3c86-4c43-8c40-b02c90413a19","Type":"ContainerDied","Data":"13039c2dba5cafdc54fa1c1db97b3fe7f2aed96d4770979b5a3f4c3122c575a1"}
Nov 02 10:38:36.722464 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:36.722391    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13039c2dba5cafdc54fa1c1db97b3fe7f2aed96d4770979b5a3f4c3122c575a1"
Nov 02 10:38:43.486304 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.486271    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"]
Nov 02 10:38:43.486883 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.486676    4449 cpu_manager.go:414] "containerMap: removing container" podUID="e95f40d2-3c86-4c43-8c40-b02c90413a19" containerName="revoke-keys"
Nov 02 10:38:43.486883 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.486686    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95f40d2-3c86-4c43-8c40-b02c90413a19" containerName="revoke-keys"
Nov 02 10:38:43.486883 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.486755    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95f40d2-3c86-4c43-8c40-b02c90413a19" containerName="revoke-keys"
Nov 02 10:38:43.490022 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.490003    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"
Nov 02 10:38:43.497666 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.497640    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:38:43.497666 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.497658    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:38:43.497851 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.497644    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:38:43.498564 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.498294    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:38:43.500116 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.500093    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"]
Nov 02 10:38:43.630538 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.630510    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dc73283-blq7w\" (UID: \"89dbb619-6a43-4d72-9181-513a5673bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"
Nov 02 10:38:43.630538 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.630539    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr6jc\" (UniqueName: \"kubernetes.io/projected/89dbb619-6a43-4d72-9181-513a5673bd67-kube-api-access-qr6jc\") pod \"maas-api-revoke-sub-0dc73283-blq7w\" (UID: \"89dbb619-6a43-4d72-9181-513a5673bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"
Nov 02 10:38:43.630738 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.630648    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dc73283-blq7w\" (UID: \"89dbb619-6a43-4d72-9181-513a5673bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"
Nov 02 10:38:43.731401 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.731379    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dc73283-blq7w\" (UID: \"89dbb619-6a43-4d72-9181-513a5673bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"
Nov 02 10:38:43.731499 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.731405    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr6jc\" (UniqueName: \"kubernetes.io/projected/89dbb619-6a43-4d72-9181-513a5673bd67-kube-api-access-qr6jc\") pod \"maas-api-revoke-sub-0dc73283-blq7w\" (UID: \"89dbb619-6a43-4d72-9181-513a5673bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"
Nov 02 10:38:43.731499 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.731462    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dc73283-blq7w\" (UID: \"89dbb619-6a43-4d72-9181-513a5673bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"
Nov 02 10:38:43.731951 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.731927    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dc73283-blq7w\" (UID: \"89dbb619-6a43-4d72-9181-513a5673bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"
Nov 02 10:38:43.733782 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.733763    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dc73283-blq7w\" (UID: \"89dbb619-6a43-4d72-9181-513a5673bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"
Nov 02 10:38:43.744195 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.744135    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr6jc\" (UniqueName: \"kubernetes.io/projected/89dbb619-6a43-4d72-9181-513a5673bd67-kube-api-access-qr6jc\") pod \"maas-api-revoke-sub-0dc73283-blq7w\" (UID: \"89dbb619-6a43-4d72-9181-513a5673bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"
Nov 02 10:38:43.804494 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:43.804472    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"
Nov 02 10:38:44.078509 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:44.078488    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"]
Nov 02 10:38:44.080871 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:38:44.080846    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89dbb619_6a43_4d72_9181_513a5673bd67.slice/crio-1c7f4e45ef02083d3e42a8e361593199f7303ed6e38fec41cc6a3f5d4b3a7639 WatchSource:0}: Error finding container 1c7f4e45ef02083d3e42a8e361593199f7303ed6e38fec41cc6a3f5d4b3a7639: Status 404 returned error can't find the container with id 1c7f4e45ef02083d3e42a8e361593199f7303ed6e38fec41cc6a3f5d4b3a7639
Nov 02 10:38:44.763990 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:44.763914    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w" event={"ID":"89dbb619-6a43-4d72-9181-513a5673bd67","Type":"ContainerStarted","Data":"a396bc3c5afbe1a85099cafaec8f86c6bb28b31d9d8ba5c4279ab64f324d9abf"}
Nov 02 10:38:44.763990 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:44.763970    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w" event={"ID":"89dbb619-6a43-4d72-9181-513a5673bd67","Type":"ContainerStarted","Data":"1c7f4e45ef02083d3e42a8e361593199f7303ed6e38fec41cc6a3f5d4b3a7639"}
Nov 02 10:38:44.785248 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:44.785194    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w" podStartSLOduration=1.785182569 podStartE2EDuration="1.785182569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:38:44.784946044 +0000 UTC m=+3801.723232819" watchObservedRunningTime="2026-11-02 10:38:44.785182569 +0000 UTC m=+3801.723469342"
Nov 02 10:38:45.773749 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:45.773709    4449 generic.go:356] "Generic (PLEG): container finished" podID="89dbb619-6a43-4d72-9181-513a5673bd67" containerID="a396bc3c5afbe1a85099cafaec8f86c6bb28b31d9d8ba5c4279ab64f324d9abf" exitCode=0
Nov 02 10:38:45.773749 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:45.773734    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w" event={"ID":"89dbb619-6a43-4d72-9181-513a5673bd67","Type":"ContainerDied","Data":"a396bc3c5afbe1a85099cafaec8f86c6bb28b31d9d8ba5c4279ab64f324d9abf"}
Nov 02 10:38:46.909580 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:46.909550    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"
Nov 02 10:38:47.061140 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:47.061119    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-openshift-ca\") pod \"89dbb619-6a43-4d72-9181-513a5673bd67\" (UID: \"89dbb619-6a43-4d72-9181-513a5673bd67\") "
Nov 02 10:38:47.061277 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:47.061172    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-xks-ca\") pod \"89dbb619-6a43-4d72-9181-513a5673bd67\" (UID: \"89dbb619-6a43-4d72-9181-513a5673bd67\") "
Nov 02 10:38:47.061277 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:47.061198    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89dbb619-6a43-4d72-9181-513a5673bd67-kube-api-access-qr6jc\" (UniqueName: \"kubernetes.io/projected/89dbb619-6a43-4d72-9181-513a5673bd67-kube-api-access-qr6jc\") pod \"89dbb619-6a43-4d72-9181-513a5673bd67\" (UID: \"89dbb619-6a43-4d72-9181-513a5673bd67\") "
Nov 02 10:38:47.061531 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:47.061505    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-openshift-ca" pod "89dbb619-6a43-4d72-9181-513a5673bd67" (UID: "89dbb619-6a43-4d72-9181-513a5673bd67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:38:47.063197 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:47.063177    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-xks-ca" pod "89dbb619-6a43-4d72-9181-513a5673bd67" (UID: "89dbb619-6a43-4d72-9181-513a5673bd67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:38:47.063336 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:47.063317    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89dbb619-6a43-4d72-9181-513a5673bd67-kube-api-access-qr6jc" pod "89dbb619-6a43-4d72-9181-513a5673bd67" (UID: "89dbb619-6a43-4d72-9181-513a5673bd67"). InnerVolumeSpecName "kube-api-access-qr6jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:38:47.162569 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:47.162543    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:47.162569 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:47.162566    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89dbb619-6a43-4d72-9181-513a5673bd67-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:47.162700 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:47.162580    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr6jc\" (UniqueName: \"kubernetes.io/projected/89dbb619-6a43-4d72-9181-513a5673bd67-kube-api-access-qr6jc\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:38:47.785417 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:47.785390    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"
Nov 02 10:38:47.785551 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:47.785420    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w" event={"ID":"89dbb619-6a43-4d72-9181-513a5673bd67","Type":"ContainerDied","Data":"1c7f4e45ef02083d3e42a8e361593199f7303ed6e38fec41cc6a3f5d4b3a7639"}
Nov 02 10:38:47.785551 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:38:47.785487    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c7f4e45ef02083d3e42a8e361593199f7303ed6e38fec41cc6a3f5d4b3a7639"
Nov 02 10:39:04.025094 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.025062    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"]
Nov 02 10:39:04.025782 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.025761    4449 cpu_manager.go:414] "containerMap: removing container" podUID="89dbb619-6a43-4d72-9181-513a5673bd67" containerName="revoke-keys"
Nov 02 10:39:04.025868 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.025785    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89dbb619-6a43-4d72-9181-513a5673bd67" containerName="revoke-keys"
Nov 02 10:39:04.025928 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.025910    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="89dbb619-6a43-4d72-9181-513a5673bd67" containerName="revoke-keys"
Nov 02 10:39:04.029101 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.029081    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"
Nov 02 10:39:04.034421 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.034398    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:39:04.035623 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.035605    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:39:04.035761 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.035685    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:39:04.035833 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.035779    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:39:04.038189 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.038163    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"]
Nov 02 10:39:04.173903 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.173874    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8tjb\" (UniqueName: \"kubernetes.io/projected/11a344ae-7697-4999-938b-776c67f7ed14-kube-api-access-q8tjb\") pod \"maas-api-revoke-sub-0b064c2b-2kjgv\" (UID: \"11a344ae-7697-4999-938b-776c67f7ed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"
Nov 02 10:39:04.174103 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.174051    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b064c2b-2kjgv\" (UID: \"11a344ae-7697-4999-938b-776c67f7ed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"
Nov 02 10:39:04.174103 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.174082    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b064c2b-2kjgv\" (UID: \"11a344ae-7697-4999-938b-776c67f7ed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"
Nov 02 10:39:04.275572 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.275495    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8tjb\" (UniqueName: \"kubernetes.io/projected/11a344ae-7697-4999-938b-776c67f7ed14-kube-api-access-q8tjb\") pod \"maas-api-revoke-sub-0b064c2b-2kjgv\" (UID: \"11a344ae-7697-4999-938b-776c67f7ed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"
Nov 02 10:39:04.275724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.275611    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b064c2b-2kjgv\" (UID: \"11a344ae-7697-4999-938b-776c67f7ed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"
Nov 02 10:39:04.275724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.275635    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b064c2b-2kjgv\" (UID: \"11a344ae-7697-4999-938b-776c67f7ed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"
Nov 02 10:39:04.276332 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.276310    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b064c2b-2kjgv\" (UID: \"11a344ae-7697-4999-938b-776c67f7ed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"
Nov 02 10:39:04.278085 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.278064    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b064c2b-2kjgv\" (UID: \"11a344ae-7697-4999-938b-776c67f7ed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"
Nov 02 10:39:04.290755 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.290722    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8tjb\" (UniqueName: \"kubernetes.io/projected/11a344ae-7697-4999-938b-776c67f7ed14-kube-api-access-q8tjb\") pod \"maas-api-revoke-sub-0b064c2b-2kjgv\" (UID: \"11a344ae-7697-4999-938b-776c67f7ed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"
Nov 02 10:39:04.345324 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.345289    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"
Nov 02 10:39:04.620013 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.619986    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"]
Nov 02 10:39:04.620976 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:39:04.620950    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a344ae_7697_4999_938b_776c67f7ed14.slice/crio-97ab86ec2922fbc0fe58267e66e8fec7eda8a1169422931cba857775e6dbb370 WatchSource:0}: Error finding container 97ab86ec2922fbc0fe58267e66e8fec7eda8a1169422931cba857775e6dbb370: Status 404 returned error can't find the container with id 97ab86ec2922fbc0fe58267e66e8fec7eda8a1169422931cba857775e6dbb370
Nov 02 10:39:04.938758 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:04.938684    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv" event={"ID":"11a344ae-7697-4999-938b-776c67f7ed14","Type":"ContainerStarted","Data":"97ab86ec2922fbc0fe58267e66e8fec7eda8a1169422931cba857775e6dbb370"}
Nov 02 10:39:05.944589 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:05.944561    4449 generic.go:356] "Generic (PLEG): container finished" podID="11a344ae-7697-4999-938b-776c67f7ed14" containerID="25ade4d7f0ac35a99a47ae30252309ee4f5189b023b873dd28ef3baa16087f85" exitCode=0
Nov 02 10:39:05.944916 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:05.944612    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv" event={"ID":"11a344ae-7697-4999-938b-776c67f7ed14","Type":"ContainerDied","Data":"25ade4d7f0ac35a99a47ae30252309ee4f5189b023b873dd28ef3baa16087f85"}
Nov 02 10:39:06.544922 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:39:06.544892    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:39:07.081627 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:07.081515    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"
Nov 02 10:39:07.201884 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:07.201830    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-xks-ca\") pod \"11a344ae-7697-4999-938b-776c67f7ed14\" (UID: \"11a344ae-7697-4999-938b-776c67f7ed14\") "
Nov 02 10:39:07.201884 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:07.201875    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a344ae-7697-4999-938b-776c67f7ed14-kube-api-access-q8tjb\" (UniqueName: \"kubernetes.io/projected/11a344ae-7697-4999-938b-776c67f7ed14-kube-api-access-q8tjb\") pod \"11a344ae-7697-4999-938b-776c67f7ed14\" (UID: \"11a344ae-7697-4999-938b-776c67f7ed14\") "
Nov 02 10:39:07.202039 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:07.201965    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-openshift-ca\") pod \"11a344ae-7697-4999-938b-776c67f7ed14\" (UID: \"11a344ae-7697-4999-938b-776c67f7ed14\") "
Nov 02 10:39:07.202337 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:07.202315    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-openshift-ca" pod "11a344ae-7697-4999-938b-776c67f7ed14" (UID: "11a344ae-7697-4999-938b-776c67f7ed14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:39:07.209275 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:07.207564    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-xks-ca" pod "11a344ae-7697-4999-938b-776c67f7ed14" (UID: "11a344ae-7697-4999-938b-776c67f7ed14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:39:07.217113 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:07.217082    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a344ae-7697-4999-938b-776c67f7ed14-kube-api-access-q8tjb" pod "11a344ae-7697-4999-938b-776c67f7ed14" (UID: "11a344ae-7697-4999-938b-776c67f7ed14"). InnerVolumeSpecName "kube-api-access-q8tjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:39:07.303289 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:07.303266    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8tjb\" (UniqueName: \"kubernetes.io/projected/11a344ae-7697-4999-938b-776c67f7ed14-kube-api-access-q8tjb\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:39:07.303289 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:07.303288    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:39:07.303415 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:07.303298    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a344ae-7697-4999-938b-776c67f7ed14-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:39:07.956678 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:07.956658    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"
Nov 02 10:39:07.956678 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:07.956666    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv" event={"ID":"11a344ae-7697-4999-938b-776c67f7ed14","Type":"ContainerDied","Data":"97ab86ec2922fbc0fe58267e66e8fec7eda8a1169422931cba857775e6dbb370"}
Nov 02 10:39:07.956848 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:07.956710    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97ab86ec2922fbc0fe58267e66e8fec7eda8a1169422931cba857775e6dbb370"
Nov 02 10:39:27.111979 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:27.111950    4449 scope.go:122] "RemoveContainer" containerID="994f354532f98fa7794866c69091c3f365c6f69d4946ea3e8bf0705036a6c14a"
Nov 02 10:39:31.715010 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.714970    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"]
Nov 02 10:39:31.715633 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.715612    4449 cpu_manager.go:414] "containerMap: removing container" podUID="11a344ae-7697-4999-938b-776c67f7ed14" containerName="revoke-keys"
Nov 02 10:39:31.715730 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.715635    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a344ae-7697-4999-938b-776c67f7ed14" containerName="revoke-keys"
Nov 02 10:39:31.715786 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.715743    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a344ae-7697-4999-938b-776c67f7ed14" containerName="revoke-keys"
Nov 02 10:39:31.718756 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.718735    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"
Nov 02 10:39:31.726239 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.726203    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:39:31.726415 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.726395    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:39:31.726415 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.726272    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:39:31.726575 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.726350    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:39:31.728770 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.728748    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"]
Nov 02 10:39:31.761971 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.761947    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btjsj\" (UniqueName: \"kubernetes.io/projected/a83b5799-8708-41ea-b6be-935bd32e8e2e-kube-api-access-btjsj\") pod \"maas-api-revoke-sub-183de081-rcwdv\" (UID: \"a83b5799-8708-41ea-b6be-935bd32e8e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"
Nov 02 10:39:31.762086 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.762072    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-183de081-rcwdv\" (UID: \"a83b5799-8708-41ea-b6be-935bd32e8e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"
Nov 02 10:39:31.762140 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.762102    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-183de081-rcwdv\" (UID: \"a83b5799-8708-41ea-b6be-935bd32e8e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"
Nov 02 10:39:31.862554 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.862530    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btjsj\" (UniqueName: \"kubernetes.io/projected/a83b5799-8708-41ea-b6be-935bd32e8e2e-kube-api-access-btjsj\") pod \"maas-api-revoke-sub-183de081-rcwdv\" (UID: \"a83b5799-8708-41ea-b6be-935bd32e8e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"
Nov 02 10:39:31.862689 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.862600    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-183de081-rcwdv\" (UID: \"a83b5799-8708-41ea-b6be-935bd32e8e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"
Nov 02 10:39:31.862689 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.862616    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-183de081-rcwdv\" (UID: \"a83b5799-8708-41ea-b6be-935bd32e8e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"
Nov 02 10:39:31.863176 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.863156    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-183de081-rcwdv\" (UID: \"a83b5799-8708-41ea-b6be-935bd32e8e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"
Nov 02 10:39:31.864996 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.864978    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-183de081-rcwdv\" (UID: \"a83b5799-8708-41ea-b6be-935bd32e8e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"
Nov 02 10:39:31.875085 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:31.875056    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btjsj\" (UniqueName: \"kubernetes.io/projected/a83b5799-8708-41ea-b6be-935bd32e8e2e-kube-api-access-btjsj\") pod \"maas-api-revoke-sub-183de081-rcwdv\" (UID: \"a83b5799-8708-41ea-b6be-935bd32e8e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"
Nov 02 10:39:32.034266 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:32.034206    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"
Nov 02 10:39:32.307924 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:32.307891    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"]
Nov 02 10:39:32.309066 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:39:32.309036    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83b5799_8708_41ea_b6be_935bd32e8e2e.slice/crio-a3bfcd3399c6ac396ebd1170b7328de3f4defcf847aaf6e0b8281ae595e32b8e WatchSource:0}: Error finding container a3bfcd3399c6ac396ebd1170b7328de3f4defcf847aaf6e0b8281ae595e32b8e: Status 404 returned error can't find the container with id a3bfcd3399c6ac396ebd1170b7328de3f4defcf847aaf6e0b8281ae595e32b8e
Nov 02 10:39:33.190705 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:33.190678    4449 generic.go:356] "Generic (PLEG): container finished" podID="a83b5799-8708-41ea-b6be-935bd32e8e2e" containerID="eaba2163e31003fb0be30dcd6bc04f4228fb3cea95bd8b99c1f095d67b71245c" exitCode=0
Nov 02 10:39:33.191001 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:33.190765    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv" event={"ID":"a83b5799-8708-41ea-b6be-935bd32e8e2e","Type":"ContainerDied","Data":"eaba2163e31003fb0be30dcd6bc04f4228fb3cea95bd8b99c1f095d67b71245c"}
Nov 02 10:39:33.191001 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:33.190817    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv" event={"ID":"a83b5799-8708-41ea-b6be-935bd32e8e2e","Type":"ContainerStarted","Data":"a3bfcd3399c6ac396ebd1170b7328de3f4defcf847aaf6e0b8281ae595e32b8e"}
Nov 02 10:39:34.324195 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:34.324172    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"
Nov 02 10:39:34.381334 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:34.381306    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-openshift-ca\") pod \"a83b5799-8708-41ea-b6be-935bd32e8e2e\" (UID: \"a83b5799-8708-41ea-b6be-935bd32e8e2e\") "
Nov 02 10:39:34.381477 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:34.381339    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-xks-ca\") pod \"a83b5799-8708-41ea-b6be-935bd32e8e2e\" (UID: \"a83b5799-8708-41ea-b6be-935bd32e8e2e\") "
Nov 02 10:39:34.381477 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:34.381413    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83b5799-8708-41ea-b6be-935bd32e8e2e-kube-api-access-btjsj\" (UniqueName: \"kubernetes.io/projected/a83b5799-8708-41ea-b6be-935bd32e8e2e-kube-api-access-btjsj\") pod \"a83b5799-8708-41ea-b6be-935bd32e8e2e\" (UID: \"a83b5799-8708-41ea-b6be-935bd32e8e2e\") "
Nov 02 10:39:34.381710 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:34.381676    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-openshift-ca" pod "a83b5799-8708-41ea-b6be-935bd32e8e2e" (UID: "a83b5799-8708-41ea-b6be-935bd32e8e2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:39:34.381862 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:34.381842    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:39:34.383472 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:34.383446    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-xks-ca" pod "a83b5799-8708-41ea-b6be-935bd32e8e2e" (UID: "a83b5799-8708-41ea-b6be-935bd32e8e2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:39:34.383590 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:34.383571    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83b5799-8708-41ea-b6be-935bd32e8e2e-kube-api-access-btjsj" pod "a83b5799-8708-41ea-b6be-935bd32e8e2e" (UID: "a83b5799-8708-41ea-b6be-935bd32e8e2e"). InnerVolumeSpecName "kube-api-access-btjsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:39:34.482682 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:34.482628    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83b5799-8708-41ea-b6be-935bd32e8e2e-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:39:34.482682 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:34.482650    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btjsj\" (UniqueName: \"kubernetes.io/projected/a83b5799-8708-41ea-b6be-935bd32e8e2e-kube-api-access-btjsj\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:39:35.202658 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:35.202638    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv"
Nov 02 10:39:35.204726 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:35.204698    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183de081-rcwdv" event={"ID":"a83b5799-8708-41ea-b6be-935bd32e8e2e","Type":"ContainerDied","Data":"a3bfcd3399c6ac396ebd1170b7328de3f4defcf847aaf6e0b8281ae595e32b8e"}
Nov 02 10:39:35.204851 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:35.204736    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3bfcd3399c6ac396ebd1170b7328de3f4defcf847aaf6e0b8281ae595e32b8e"
Nov 02 10:39:36.544987 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:39:36.544952    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:39:38.036604 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:38.036432    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"]
Nov 02 10:39:38.039910 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:38.039877    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80d7a79e-g6b7q"]
Nov 02 10:39:39.205092 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:39.205063    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f476720-58b9-4bae-993f-b3f300ea65e9" path="/var/lib/kubelet/pods/5f476720-58b9-4bae-993f-b3f300ea65e9/volumes"
Nov 02 10:39:41.992467 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:41.992414    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"]
Nov 02 10:39:41.992901 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:41.992887    4449 cpu_manager.go:414] "containerMap: removing container" podUID="a83b5799-8708-41ea-b6be-935bd32e8e2e" containerName="revoke-keys"
Nov 02 10:39:41.992946 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:41.992901    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83b5799-8708-41ea-b6be-935bd32e8e2e" containerName="revoke-keys"
Nov 02 10:39:41.992990 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:41.992979    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83b5799-8708-41ea-b6be-935bd32e8e2e" containerName="revoke-keys"
Nov 02 10:39:41.997226 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:41.997197    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"
Nov 02 10:39:42.001829 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.001590    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:39:42.002381 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.002357    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:39:42.014272 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.003189    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:39:42.014272 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.003649    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:39:42.014272 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.012113    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"]
Nov 02 10:39:42.049033 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.049011    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6785a9e6-pwb44\" (UID: \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"
Nov 02 10:39:42.049252 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.049054    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfxxf\" (UniqueName: \"kubernetes.io/projected/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-kube-api-access-xfxxf\") pod \"maas-api-revoke-sub-6785a9e6-pwb44\" (UID: \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"
Nov 02 10:39:42.049252 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.049136    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6785a9e6-pwb44\" (UID: \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"
Nov 02 10:39:42.150245 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.150206    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6785a9e6-pwb44\" (UID: \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"
Nov 02 10:39:42.150340 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.150271    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfxxf\" (UniqueName: \"kubernetes.io/projected/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-kube-api-access-xfxxf\") pod \"maas-api-revoke-sub-6785a9e6-pwb44\" (UID: \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"
Nov 02 10:39:42.150340 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.150321    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6785a9e6-pwb44\" (UID: \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"
Nov 02 10:39:42.150927 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.150909    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6785a9e6-pwb44\" (UID: \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"
Nov 02 10:39:42.152686 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.152659    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6785a9e6-pwb44\" (UID: \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"
Nov 02 10:39:42.162617 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.162595    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfxxf\" (UniqueName: \"kubernetes.io/projected/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-kube-api-access-xfxxf\") pod \"maas-api-revoke-sub-6785a9e6-pwb44\" (UID: \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"
Nov 02 10:39:42.323082 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.323062    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"
Nov 02 10:39:42.596263 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:42.596240    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"]
Nov 02 10:39:42.598293 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:39:42.598268    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aac4725_b768_4e0b_b4b3_fd7a0c5542d0.slice/crio-ebd4ef1ce777ba3b80212dc5873fe60ec68b8f271229d14b0b8eac5cf9ffb127 WatchSource:0}: Error finding container ebd4ef1ce777ba3b80212dc5873fe60ec68b8f271229d14b0b8eac5cf9ffb127: Status 404 returned error can't find the container with id ebd4ef1ce777ba3b80212dc5873fe60ec68b8f271229d14b0b8eac5cf9ffb127
Nov 02 10:39:43.245193 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:43.245106    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44" event={"ID":"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0","Type":"ContainerStarted","Data":"9195a3f3d273d9a7b6764b0af8deaace1d3b3f56e9f096156701f14ba602055d"}
Nov 02 10:39:43.245193 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:43.245152    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44" event={"ID":"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0","Type":"ContainerStarted","Data":"ebd4ef1ce777ba3b80212dc5873fe60ec68b8f271229d14b0b8eac5cf9ffb127"}
Nov 02 10:39:43.265313 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:43.265273    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44" podStartSLOduration=2.265259947 podStartE2EDuration="2.265259947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:39:43.263030162 +0000 UTC m=+3860.201316936" watchObservedRunningTime="2026-11-02 10:39:43.265259947 +0000 UTC m=+3860.203546722"
Nov 02 10:39:44.251974 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:44.251947    4449 generic.go:356] "Generic (PLEG): container finished" podID="2aac4725-b768-4e0b-b4b3-fd7a0c5542d0" containerID="9195a3f3d273d9a7b6764b0af8deaace1d3b3f56e9f096156701f14ba602055d" exitCode=0
Nov 02 10:39:44.252385 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:44.252034    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44" event={"ID":"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0","Type":"ContainerDied","Data":"9195a3f3d273d9a7b6764b0af8deaace1d3b3f56e9f096156701f14ba602055d"}
Nov 02 10:39:45.385036 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:45.385008    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"
Nov 02 10:39:45.479011 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:45.478985    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-openshift-ca\") pod \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\" (UID: \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\") "
Nov 02 10:39:45.479150 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:45.479046    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-xks-ca\") pod \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\" (UID: \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\") "
Nov 02 10:39:45.479150 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:45.479080    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-kube-api-access-xfxxf\" (UniqueName: \"kubernetes.io/projected/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-kube-api-access-xfxxf\") pod \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\" (UID: \"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0\") "
Nov 02 10:39:45.479344 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:45.479321    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-openshift-ca" pod "2aac4725-b768-4e0b-b4b3-fd7a0c5542d0" (UID: "2aac4725-b768-4e0b-b4b3-fd7a0c5542d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:39:45.479461 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:45.479443    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:39:45.481095 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:45.481073    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-xks-ca" pod "2aac4725-b768-4e0b-b4b3-fd7a0c5542d0" (UID: "2aac4725-b768-4e0b-b4b3-fd7a0c5542d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:39:45.481193 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:45.481126    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-kube-api-access-xfxxf" pod "2aac4725-b768-4e0b-b4b3-fd7a0c5542d0" (UID: "2aac4725-b768-4e0b-b4b3-fd7a0c5542d0"). InnerVolumeSpecName "kube-api-access-xfxxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:39:45.580636 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:45.580616    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:39:45.580736 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:45.580637    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfxxf\" (UniqueName: \"kubernetes.io/projected/2aac4725-b768-4e0b-b4b3-fd7a0c5542d0-kube-api-access-xfxxf\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:39:46.263597 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:46.263575    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44"
Nov 02 10:39:46.263725 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:46.263577    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6785a9e6-pwb44" event={"ID":"2aac4725-b768-4e0b-b4b3-fd7a0c5542d0","Type":"ContainerDied","Data":"ebd4ef1ce777ba3b80212dc5873fe60ec68b8f271229d14b0b8eac5cf9ffb127"}
Nov 02 10:39:46.263788 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:46.263720    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd4ef1ce777ba3b80212dc5873fe60ec68b8f271229d14b0b8eac5cf9ffb127"
Nov 02 10:39:58.738342 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:58.738304    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:39:58.738938 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:39:58.738904    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:40:06.661427 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:40:06.661386    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:40:07.025295 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:07.024311    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:40:07.025295 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:07.024373    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j" podUID="51e601b5-d915-4326-b964-523a4f66788d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:40:07.059243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:07.058297    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6cftx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:40:07.059243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:07.058384    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx" podUID="604d7b4e-efdd-4979-b544-fd3d4d1953a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:40:10.943273 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:10.943241    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"]
Nov 02 10:40:10.943704 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:10.943691    4449 cpu_manager.go:414] "containerMap: removing container" podUID="2aac4725-b768-4e0b-b4b3-fd7a0c5542d0" containerName="revoke-keys"
Nov 02 10:40:10.943751 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:10.943705    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aac4725-b768-4e0b-b4b3-fd7a0c5542d0" containerName="revoke-keys"
Nov 02 10:40:10.943803 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:10.943794    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aac4725-b768-4e0b-b4b3-fd7a0c5542d0" containerName="revoke-keys"
Nov 02 10:40:10.947100 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:10.947080    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"
Nov 02 10:40:10.953178 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:10.953155    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:40:10.953405 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:10.953304    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:40:10.953522 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:10.953469    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:40:10.953599 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:10.953540    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:40:10.956127 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:10.956104    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"]
Nov 02 10:40:11.080807 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:11.080773    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96pfg\" (UniqueName: \"kubernetes.io/projected/5a628840-a776-4725-86f5-c2e153808300-kube-api-access-96pfg\") pod \"maas-api-revoke-sub-fe5e5cc3-85k7k\" (UID: \"5a628840-a776-4725-86f5-c2e153808300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"
Nov 02 10:40:11.081005 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:11.080830    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a628840-a776-4725-86f5-c2e153808300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe5e5cc3-85k7k\" (UID: \"5a628840-a776-4725-86f5-c2e153808300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"
Nov 02 10:40:11.081085 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:11.081057    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a628840-a776-4725-86f5-c2e153808300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe5e5cc3-85k7k\" (UID: \"5a628840-a776-4725-86f5-c2e153808300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"
Nov 02 10:40:11.181888 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:11.181862    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a628840-a776-4725-86f5-c2e153808300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe5e5cc3-85k7k\" (UID: \"5a628840-a776-4725-86f5-c2e153808300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"
Nov 02 10:40:11.182038 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:11.181920    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96pfg\" (UniqueName: \"kubernetes.io/projected/5a628840-a776-4725-86f5-c2e153808300-kube-api-access-96pfg\") pod \"maas-api-revoke-sub-fe5e5cc3-85k7k\" (UID: \"5a628840-a776-4725-86f5-c2e153808300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"
Nov 02 10:40:11.182038 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:11.181947    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a628840-a776-4725-86f5-c2e153808300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe5e5cc3-85k7k\" (UID: \"5a628840-a776-4725-86f5-c2e153808300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"
Nov 02 10:40:11.182507 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:11.182478    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a628840-a776-4725-86f5-c2e153808300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe5e5cc3-85k7k\" (UID: \"5a628840-a776-4725-86f5-c2e153808300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"
Nov 02 10:40:11.184295 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:11.184275    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a628840-a776-4725-86f5-c2e153808300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe5e5cc3-85k7k\" (UID: \"5a628840-a776-4725-86f5-c2e153808300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"
Nov 02 10:40:11.194641 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:11.194586    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96pfg\" (UniqueName: \"kubernetes.io/projected/5a628840-a776-4725-86f5-c2e153808300-kube-api-access-96pfg\") pod \"maas-api-revoke-sub-fe5e5cc3-85k7k\" (UID: \"5a628840-a776-4725-86f5-c2e153808300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"
Nov 02 10:40:11.265329 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:11.265305    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"
Nov 02 10:40:11.538567 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:11.538536    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"]
Nov 02 10:40:11.540117 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:40:11.540085    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a628840_a776_4725_86f5_c2e153808300.slice/crio-8fad3e2d6e1a890b9893403b3f61e616078a3d4eb51f8ba083b495e3b9be6256 WatchSource:0}: Error finding container 8fad3e2d6e1a890b9893403b3f61e616078a3d4eb51f8ba083b495e3b9be6256: Status 404 returned error can't find the container with id 8fad3e2d6e1a890b9893403b3f61e616078a3d4eb51f8ba083b495e3b9be6256
Nov 02 10:40:12.529210 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:12.529147    4449 generic.go:356] "Generic (PLEG): container finished" podID="5a628840-a776-4725-86f5-c2e153808300" containerID="e1f55bd987ec3209ebafe86946a6e392958eac5c97864c9ddc65fea0df7e5a75" exitCode=0
Nov 02 10:40:12.529210 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:12.529201    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k" event={"ID":"5a628840-a776-4725-86f5-c2e153808300","Type":"ContainerDied","Data":"e1f55bd987ec3209ebafe86946a6e392958eac5c97864c9ddc65fea0df7e5a75"}
Nov 02 10:40:12.529651 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:12.529261    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k" event={"ID":"5a628840-a776-4725-86f5-c2e153808300","Type":"ContainerStarted","Data":"8fad3e2d6e1a890b9893403b3f61e616078a3d4eb51f8ba083b495e3b9be6256"}
Nov 02 10:40:13.667833 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:13.667802    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"
Nov 02 10:40:13.807868 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:13.807813    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a628840-a776-4725-86f5-c2e153808300-kube-api-access-96pfg\" (UniqueName: \"kubernetes.io/projected/5a628840-a776-4725-86f5-c2e153808300-kube-api-access-96pfg\") pod \"5a628840-a776-4725-86f5-c2e153808300\" (UID: \"5a628840-a776-4725-86f5-c2e153808300\") "
Nov 02 10:40:13.807868 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:13.807844    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a628840-a776-4725-86f5-c2e153808300-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a628840-a776-4725-86f5-c2e153808300-maas-api-xks-ca\") pod \"5a628840-a776-4725-86f5-c2e153808300\" (UID: \"5a628840-a776-4725-86f5-c2e153808300\") "
Nov 02 10:40:13.808029 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:13.807881    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a628840-a776-4725-86f5-c2e153808300-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a628840-a776-4725-86f5-c2e153808300-maas-api-openshift-ca\") pod \"5a628840-a776-4725-86f5-c2e153808300\" (UID: \"5a628840-a776-4725-86f5-c2e153808300\") "
Nov 02 10:40:13.808372 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:13.808342    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a628840-a776-4725-86f5-c2e153808300-maas-api-openshift-ca" pod "5a628840-a776-4725-86f5-c2e153808300" (UID: "5a628840-a776-4725-86f5-c2e153808300"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:40:13.810006 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:13.809987    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a628840-a776-4725-86f5-c2e153808300-maas-api-xks-ca" pod "5a628840-a776-4725-86f5-c2e153808300" (UID: "5a628840-a776-4725-86f5-c2e153808300"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:40:13.810102 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:13.810068    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a628840-a776-4725-86f5-c2e153808300-kube-api-access-96pfg" pod "5a628840-a776-4725-86f5-c2e153808300" (UID: "5a628840-a776-4725-86f5-c2e153808300"). InnerVolumeSpecName "kube-api-access-96pfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:40:13.909082 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:13.909050    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a628840-a776-4725-86f5-c2e153808300-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:40:13.909196 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:13.909083    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96pfg\" (UniqueName: \"kubernetes.io/projected/5a628840-a776-4725-86f5-c2e153808300-kube-api-access-96pfg\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:40:13.909196 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:13.909097    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a628840-a776-4725-86f5-c2e153808300-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:40:14.541516 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:14.541495    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k" event={"ID":"5a628840-a776-4725-86f5-c2e153808300","Type":"ContainerDied","Data":"8fad3e2d6e1a890b9893403b3f61e616078a3d4eb51f8ba083b495e3b9be6256"}
Nov 02 10:40:14.541650 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:14.541532    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fad3e2d6e1a890b9893403b3f61e616078a3d4eb51f8ba083b495e3b9be6256"
Nov 02 10:40:14.541650 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:14.541503    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5e5cc3-85k7k"
Nov 02 10:40:21.168954 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.168925    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"]
Nov 02 10:40:21.169376 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.169367    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5a628840-a776-4725-86f5-c2e153808300" containerName="revoke-keys"
Nov 02 10:40:21.169430 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.169378    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a628840-a776-4725-86f5-c2e153808300" containerName="revoke-keys"
Nov 02 10:40:21.169466 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.169453    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a628840-a776-4725-86f5-c2e153808300" containerName="revoke-keys"
Nov 02 10:40:21.172794 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.172775    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"
Nov 02 10:40:21.182017 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.181991    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"]
Nov 02 10:40:21.184419 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.184394    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:40:21.184419 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.184398    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:40:21.184617 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.184428    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:40:21.184617 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.184527    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:40:21.265197 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.265169    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxqn6\" (UniqueName: \"kubernetes.io/projected/f03598b8-71e2-49f4-a904-dcbd3afa85db-kube-api-access-gxqn6\") pod \"maas-api-revoke-sub-2b225abb-cq4fc\" (UID: \"f03598b8-71e2-49f4-a904-dcbd3afa85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"
Nov 02 10:40:21.265357 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.265244    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b225abb-cq4fc\" (UID: \"f03598b8-71e2-49f4-a904-dcbd3afa85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"
Nov 02 10:40:21.265357 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.265333    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b225abb-cq4fc\" (UID: \"f03598b8-71e2-49f4-a904-dcbd3afa85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"
Nov 02 10:40:21.365717 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.365689    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqn6\" (UniqueName: \"kubernetes.io/projected/f03598b8-71e2-49f4-a904-dcbd3afa85db-kube-api-access-gxqn6\") pod \"maas-api-revoke-sub-2b225abb-cq4fc\" (UID: \"f03598b8-71e2-49f4-a904-dcbd3afa85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"
Nov 02 10:40:21.365852 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.365750    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b225abb-cq4fc\" (UID: \"f03598b8-71e2-49f4-a904-dcbd3afa85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"
Nov 02 10:40:21.365852 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.365794    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b225abb-cq4fc\" (UID: \"f03598b8-71e2-49f4-a904-dcbd3afa85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"
Nov 02 10:40:21.366481 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.366463    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b225abb-cq4fc\" (UID: \"f03598b8-71e2-49f4-a904-dcbd3afa85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"
Nov 02 10:40:21.368104 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.368087    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b225abb-cq4fc\" (UID: \"f03598b8-71e2-49f4-a904-dcbd3afa85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"
Nov 02 10:40:21.378854 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.378831    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxqn6\" (UniqueName: \"kubernetes.io/projected/f03598b8-71e2-49f4-a904-dcbd3afa85db-kube-api-access-gxqn6\") pod \"maas-api-revoke-sub-2b225abb-cq4fc\" (UID: \"f03598b8-71e2-49f4-a904-dcbd3afa85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"
Nov 02 10:40:21.492398 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.492352    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"
Nov 02 10:40:21.767721 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:21.767699    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"]
Nov 02 10:40:21.769087 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:40:21.769059    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03598b8_71e2_49f4_a904_dcbd3afa85db.slice/crio-be5a6d7355862be154bb2f360908aedbf5ead4ca764c1f1e2cc407f2435f88c6 WatchSource:0}: Error finding container be5a6d7355862be154bb2f360908aedbf5ead4ca764c1f1e2cc407f2435f88c6: Status 404 returned error can't find the container with id be5a6d7355862be154bb2f360908aedbf5ead4ca764c1f1e2cc407f2435f88c6
Nov 02 10:40:22.588472 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:22.588411    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc" event={"ID":"f03598b8-71e2-49f4-a904-dcbd3afa85db","Type":"ContainerStarted","Data":"a004aeb87f301bd0c0987c011f0293fdac13f67ed78491175191eaea8af90c3c"}
Nov 02 10:40:22.588978 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:22.588501    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc" event={"ID":"f03598b8-71e2-49f4-a904-dcbd3afa85db","Type":"ContainerStarted","Data":"be5a6d7355862be154bb2f360908aedbf5ead4ca764c1f1e2cc407f2435f88c6"}
Nov 02 10:40:22.609370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:22.609307    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc" podStartSLOduration=1.609290289 podStartE2EDuration="1.609290289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:40:22.607319268 +0000 UTC m=+3899.545606067" watchObservedRunningTime="2026-11-02 10:40:22.609290289 +0000 UTC m=+3899.547577064"
Nov 02 10:40:23.639946 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:23.639909    4449 generic.go:356] "Generic (PLEG): container finished" podID="f03598b8-71e2-49f4-a904-dcbd3afa85db" containerID="a004aeb87f301bd0c0987c011f0293fdac13f67ed78491175191eaea8af90c3c" exitCode=0
Nov 02 10:40:23.642677 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:23.642647    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc" event={"ID":"f03598b8-71e2-49f4-a904-dcbd3afa85db","Type":"ContainerDied","Data":"a004aeb87f301bd0c0987c011f0293fdac13f67ed78491175191eaea8af90c3c"}
Nov 02 10:40:25.353511 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:25.353468    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"
Nov 02 10:40:25.534589 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:25.534552    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-openshift-ca\") pod \"f03598b8-71e2-49f4-a904-dcbd3afa85db\" (UID: \"f03598b8-71e2-49f4-a904-dcbd3afa85db\") "
Nov 02 10:40:25.534901 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:25.534878    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-xks-ca\") pod \"f03598b8-71e2-49f4-a904-dcbd3afa85db\" (UID: \"f03598b8-71e2-49f4-a904-dcbd3afa85db\") "
Nov 02 10:40:25.535054 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:25.535038    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03598b8-71e2-49f4-a904-dcbd3afa85db-kube-api-access-gxqn6\" (UniqueName: \"kubernetes.io/projected/f03598b8-71e2-49f4-a904-dcbd3afa85db-kube-api-access-gxqn6\") pod \"f03598b8-71e2-49f4-a904-dcbd3afa85db\" (UID: \"f03598b8-71e2-49f4-a904-dcbd3afa85db\") "
Nov 02 10:40:25.536727 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:25.536701    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-openshift-ca" pod "f03598b8-71e2-49f4-a904-dcbd3afa85db" (UID: "f03598b8-71e2-49f4-a904-dcbd3afa85db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:40:25.540718 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:25.540688    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-xks-ca" pod "f03598b8-71e2-49f4-a904-dcbd3afa85db" (UID: "f03598b8-71e2-49f4-a904-dcbd3afa85db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:40:25.545700 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:25.545648    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03598b8-71e2-49f4-a904-dcbd3afa85db-kube-api-access-gxqn6" pod "f03598b8-71e2-49f4-a904-dcbd3afa85db" (UID: "f03598b8-71e2-49f4-a904-dcbd3afa85db"). InnerVolumeSpecName "kube-api-access-gxqn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:40:25.644924 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:25.644890    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxqn6\" (UniqueName: \"kubernetes.io/projected/f03598b8-71e2-49f4-a904-dcbd3afa85db-kube-api-access-gxqn6\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:40:25.645343 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:25.645324    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:40:25.645526 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:25.645508    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03598b8-71e2-49f4-a904-dcbd3afa85db-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:40:25.686244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:25.679818    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc" event={"ID":"f03598b8-71e2-49f4-a904-dcbd3afa85db","Type":"ContainerDied","Data":"be5a6d7355862be154bb2f360908aedbf5ead4ca764c1f1e2cc407f2435f88c6"}
Nov 02 10:40:25.686244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:25.679877    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be5a6d7355862be154bb2f360908aedbf5ead4ca764c1f1e2cc407f2435f88c6"
Nov 02 10:40:25.686244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:25.680038    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b225abb-cq4fc"
Nov 02 10:40:36.668723 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:40:36.668504    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:40:36.920449 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:36.920325    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:40:36.920750 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:36.920700    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-mqzmk" podUID="de093953-9e5f-4d01-9e8f-0404f29e0b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:40:37.018397 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:37.018368    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:40:37.018688 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:37.018660    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9nd6j" podUID="51e601b5-d915-4326-b964-523a4f66788d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:40:37.050375 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:37.050348    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-6cftx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:40:37.050553 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:37.050527    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-6cftx" podUID="604d7b4e-efdd-4979-b544-fd3d4d1953a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:40:48.320243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.317288    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"]
Nov 02 10:40:48.320243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.318049    4449 cpu_manager.go:414] "containerMap: removing container" podUID="f03598b8-71e2-49f4-a904-dcbd3afa85db" containerName="revoke-keys"
Nov 02 10:40:48.320243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.318069    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03598b8-71e2-49f4-a904-dcbd3afa85db" containerName="revoke-keys"
Nov 02 10:40:48.320243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.318208    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03598b8-71e2-49f4-a904-dcbd3afa85db" containerName="revoke-keys"
Nov 02 10:40:48.323773 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.322070    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"
Nov 02 10:40:48.333300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.331433    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:40:48.333300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.332036    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:40:48.350248 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.344518    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"]
Nov 02 10:40:48.403535 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.402825    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2caa5c6d-8v5rh\" (UID: \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"
Nov 02 10:40:48.403535 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.402875    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2caa5c6d-8v5rh\" (UID: \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"
Nov 02 10:40:48.403535 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.402996    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dhk5\" (UniqueName: \"kubernetes.io/projected/c56e0d31-d49f-416f-9d10-5b3494cdd3da-kube-api-access-4dhk5\") pod \"maas-api-revoke-sub-2caa5c6d-8v5rh\" (UID: \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"
Nov 02 10:40:48.403535 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.403373    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:40:48.403535 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.403503    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:40:48.506978 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.506946    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dhk5\" (UniqueName: \"kubernetes.io/projected/c56e0d31-d49f-416f-9d10-5b3494cdd3da-kube-api-access-4dhk5\") pod \"maas-api-revoke-sub-2caa5c6d-8v5rh\" (UID: \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"
Nov 02 10:40:48.507409 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.507369    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2caa5c6d-8v5rh\" (UID: \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"
Nov 02 10:40:48.511238 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.507585    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2caa5c6d-8v5rh\" (UID: \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"
Nov 02 10:40:48.552784 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.552727    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2caa5c6d-8v5rh\" (UID: \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"
Nov 02 10:40:48.560519 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.560486    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2caa5c6d-8v5rh\" (UID: \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"
Nov 02 10:40:48.576443 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.576352    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dhk5\" (UniqueName: \"kubernetes.io/projected/c56e0d31-d49f-416f-9d10-5b3494cdd3da-kube-api-access-4dhk5\") pod \"maas-api-revoke-sub-2caa5c6d-8v5rh\" (UID: \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"
Nov 02 10:40:48.758681 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:48.758395    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"
Nov 02 10:40:49.582302 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:49.568100    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"]
Nov 02 10:40:50.008784 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:50.008752    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh" event={"ID":"c56e0d31-d49f-416f-9d10-5b3494cdd3da","Type":"ContainerStarted","Data":"4086e05ff779bbb1e5ef77a0323f6632e3e7d3f44c26b0808229eb73c8de34fc"}
Nov 02 10:40:50.008897 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:50.008836    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh" event={"ID":"c56e0d31-d49f-416f-9d10-5b3494cdd3da","Type":"ContainerStarted","Data":"d86102ee991c72d322383ab4853cf357dc6ec9441c002711aab9de5cb6cdfa69"}
Nov 02 10:40:50.026998 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:50.026961    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh" podStartSLOduration=2.0269489 podStartE2EDuration="2.0269489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:40:50.026537196 +0000 UTC m=+3926.964823972" watchObservedRunningTime="2026-11-02 10:40:50.0269489 +0000 UTC m=+3926.965235684"
Nov 02 10:40:51.015983 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:51.015956    4449 generic.go:356] "Generic (PLEG): container finished" podID="c56e0d31-d49f-416f-9d10-5b3494cdd3da" containerID="4086e05ff779bbb1e5ef77a0323f6632e3e7d3f44c26b0808229eb73c8de34fc" exitCode=0
Nov 02 10:40:51.016349 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:51.016002    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh" event={"ID":"c56e0d31-d49f-416f-9d10-5b3494cdd3da","Type":"ContainerDied","Data":"4086e05ff779bbb1e5ef77a0323f6632e3e7d3f44c26b0808229eb73c8de34fc"}
Nov 02 10:40:52.150246 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:52.150206    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"
Nov 02 10:40:52.246473 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:52.246449    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-openshift-ca\") pod \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\" (UID: \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\") "
Nov 02 10:40:52.246627 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:52.246484    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-xks-ca\") pod \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\" (UID: \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\") "
Nov 02 10:40:52.246697 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:52.246626    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56e0d31-d49f-416f-9d10-5b3494cdd3da-kube-api-access-4dhk5\" (UniqueName: \"kubernetes.io/projected/c56e0d31-d49f-416f-9d10-5b3494cdd3da-kube-api-access-4dhk5\") pod \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\" (UID: \"c56e0d31-d49f-416f-9d10-5b3494cdd3da\") "
Nov 02 10:40:52.246793 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:52.246773    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-openshift-ca" pod "c56e0d31-d49f-416f-9d10-5b3494cdd3da" (UID: "c56e0d31-d49f-416f-9d10-5b3494cdd3da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:40:52.246965 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:52.246951    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:40:52.248509 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:52.248490    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-xks-ca" pod "c56e0d31-d49f-416f-9d10-5b3494cdd3da" (UID: "c56e0d31-d49f-416f-9d10-5b3494cdd3da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:40:52.248640 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:52.248627    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56e0d31-d49f-416f-9d10-5b3494cdd3da-kube-api-access-4dhk5" pod "c56e0d31-d49f-416f-9d10-5b3494cdd3da" (UID: "c56e0d31-d49f-416f-9d10-5b3494cdd3da"). InnerVolumeSpecName "kube-api-access-4dhk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:40:52.348287 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:52.348267    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56e0d31-d49f-416f-9d10-5b3494cdd3da-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:40:52.348287 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:52.348286    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dhk5\" (UniqueName: \"kubernetes.io/projected/c56e0d31-d49f-416f-9d10-5b3494cdd3da-kube-api-access-4dhk5\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:40:53.028538 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:53.028518    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh"
Nov 02 10:40:53.028671 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:53.028538    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2caa5c6d-8v5rh" event={"ID":"c56e0d31-d49f-416f-9d10-5b3494cdd3da","Type":"ContainerDied","Data":"d86102ee991c72d322383ab4853cf357dc6ec9441c002711aab9de5cb6cdfa69"}
Nov 02 10:40:53.028671 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:53.028582    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d86102ee991c72d322383ab4853cf357dc6ec9441c002711aab9de5cb6cdfa69"
Nov 02 10:40:58.594102 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.594063    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"]
Nov 02 10:40:58.594522 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.594508    4449 cpu_manager.go:414] "containerMap: removing container" podUID="c56e0d31-d49f-416f-9d10-5b3494cdd3da" containerName="revoke-keys"
Nov 02 10:40:58.594566 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.594523    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56e0d31-d49f-416f-9d10-5b3494cdd3da" containerName="revoke-keys"
Nov 02 10:40:58.594638 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.594629    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56e0d31-d49f-416f-9d10-5b3494cdd3da" containerName="revoke-keys"
Nov 02 10:40:58.597944 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.597929    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"
Nov 02 10:40:58.607246 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.606531    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:40:58.607246 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.606572    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"]
Nov 02 10:40:58.607246 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.606649    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:40:58.607246 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.606701    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:40:58.607246 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.606721    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:40:58.703300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.703269    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd04fbcf-g4jph\" (UID: \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"
Nov 02 10:40:58.703443 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.703387    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd04fbcf-g4jph\" (UID: \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"
Nov 02 10:40:58.703485 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.703459    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p792n\" (UniqueName: \"kubernetes.io/projected/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-kube-api-access-p792n\") pod \"maas-api-revoke-sub-fd04fbcf-g4jph\" (UID: \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"
Nov 02 10:40:58.804801 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.804778    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd04fbcf-g4jph\" (UID: \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"
Nov 02 10:40:58.804935 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.804857    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd04fbcf-g4jph\" (UID: \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"
Nov 02 10:40:58.804935 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.804891    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p792n\" (UniqueName: \"kubernetes.io/projected/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-kube-api-access-p792n\") pod \"maas-api-revoke-sub-fd04fbcf-g4jph\" (UID: \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"
Nov 02 10:40:58.805607 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.805565    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd04fbcf-g4jph\" (UID: \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"
Nov 02 10:40:58.807136 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.807112    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd04fbcf-g4jph\" (UID: \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"
Nov 02 10:40:58.817876 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.817855    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p792n\" (UniqueName: \"kubernetes.io/projected/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-kube-api-access-p792n\") pod \"maas-api-revoke-sub-fd04fbcf-g4jph\" (UID: \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"
Nov 02 10:40:58.913833 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:58.913789    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"
Nov 02 10:40:59.187532 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:40:59.187492    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"]
Nov 02 10:40:59.188998 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:40:59.188976    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7e88d2_bcc5_4296_8707_4b7f7a86de96.slice/crio-f4e554f8279a8cdd203baca92f11c567cc9a63fd9060e6c4f1dd1f7c912592a4 WatchSource:0}: Error finding container f4e554f8279a8cdd203baca92f11c567cc9a63fd9060e6c4f1dd1f7c912592a4: Status 404 returned error can't find the container with id f4e554f8279a8cdd203baca92f11c567cc9a63fd9060e6c4f1dd1f7c912592a4
Nov 02 10:41:00.071314 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:00.071285    4449 generic.go:356] "Generic (PLEG): container finished" podID="5e7e88d2-bcc5-4296-8707-4b7f7a86de96" containerID="250bc5c333daa27cc7f298497d3e2380232683ac75daebe0df97df74acb279b1" exitCode=0
Nov 02 10:41:00.071651 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:00.071337    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph" event={"ID":"5e7e88d2-bcc5-4296-8707-4b7f7a86de96","Type":"ContainerDied","Data":"250bc5c333daa27cc7f298497d3e2380232683ac75daebe0df97df74acb279b1"}
Nov 02 10:41:00.071651 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:00.071387    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph" event={"ID":"5e7e88d2-bcc5-4296-8707-4b7f7a86de96","Type":"ContainerStarted","Data":"f4e554f8279a8cdd203baca92f11c567cc9a63fd9060e6c4f1dd1f7c912592a4"}
Nov 02 10:41:01.265908 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:01.265880    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"
Nov 02 10:41:01.327265 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:01.327226    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-xks-ca\") pod \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\" (UID: \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\") "
Nov 02 10:41:01.327404 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:01.327291    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-kube-api-access-p792n\" (UniqueName: \"kubernetes.io/projected/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-kube-api-access-p792n\") pod \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\" (UID: \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\") "
Nov 02 10:41:01.327404 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:01.327324    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-openshift-ca\") pod \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\" (UID: \"5e7e88d2-bcc5-4296-8707-4b7f7a86de96\") "
Nov 02 10:41:01.328156 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:01.328128    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-openshift-ca" pod "5e7e88d2-bcc5-4296-8707-4b7f7a86de96" (UID: "5e7e88d2-bcc5-4296-8707-4b7f7a86de96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:41:01.330920 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:01.330852    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-kube-api-access-p792n" pod "5e7e88d2-bcc5-4296-8707-4b7f7a86de96" (UID: "5e7e88d2-bcc5-4296-8707-4b7f7a86de96"). InnerVolumeSpecName "kube-api-access-p792n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:41:01.342918 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:01.342882    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-xks-ca" pod "5e7e88d2-bcc5-4296-8707-4b7f7a86de96" (UID: "5e7e88d2-bcc5-4296-8707-4b7f7a86de96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:41:01.434243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:01.430904    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:01.434243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:01.430952    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p792n\" (UniqueName: \"kubernetes.io/projected/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-kube-api-access-p792n\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:01.434243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:01.430971    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e7e88d2-bcc5-4296-8707-4b7f7a86de96-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:02.103244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:02.102964    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph" event={"ID":"5e7e88d2-bcc5-4296-8707-4b7f7a86de96","Type":"ContainerDied","Data":"f4e554f8279a8cdd203baca92f11c567cc9a63fd9060e6c4f1dd1f7c912592a4"}
Nov 02 10:41:02.103244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:02.103027    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4e554f8279a8cdd203baca92f11c567cc9a63fd9060e6c4f1dd1f7c912592a4"
Nov 02 10:41:02.103244 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:02.103094    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd04fbcf-g4jph"
Nov 02 10:41:06.544369 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:41:06.544334    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:41:08.797737 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:08.797708    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"]
Nov 02 10:41:08.798187 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:08.798171    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5e7e88d2-bcc5-4296-8707-4b7f7a86de96" containerName="revoke-keys"
Nov 02 10:41:08.798247 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:08.798189    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e7e88d2-bcc5-4296-8707-4b7f7a86de96" containerName="revoke-keys"
Nov 02 10:41:08.798299 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:08.798289    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e7e88d2-bcc5-4296-8707-4b7f7a86de96" containerName="revoke-keys"
Nov 02 10:41:08.801277 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:08.801259    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"
Nov 02 10:41:08.807598 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:08.807574    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:41:08.807892 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:08.807598    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:41:08.808036 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:08.807635    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:41:08.808787 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:08.808753    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:41:08.813053 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:08.813027    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"]
Nov 02 10:41:08.925922 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:08.925896    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmw2n\" (UniqueName: \"kubernetes.io/projected/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-kube-api-access-rmw2n\") pod \"maas-api-revoke-sub-85eb5cec-nm9t5\" (UID: \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"
Nov 02 10:41:08.926083 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:08.925947    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85eb5cec-nm9t5\" (UID: \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"
Nov 02 10:41:08.926154 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:08.926098    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85eb5cec-nm9t5\" (UID: \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"
Nov 02 10:41:09.026789 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:09.026765    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85eb5cec-nm9t5\" (UID: \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"
Nov 02 10:41:09.026916 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:09.026818    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmw2n\" (UniqueName: \"kubernetes.io/projected/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-kube-api-access-rmw2n\") pod \"maas-api-revoke-sub-85eb5cec-nm9t5\" (UID: \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"
Nov 02 10:41:09.026916 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:09.026853    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85eb5cec-nm9t5\" (UID: \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"
Nov 02 10:41:09.027422 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:09.027405    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85eb5cec-nm9t5\" (UID: \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"
Nov 02 10:41:09.029120 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:09.029096    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85eb5cec-nm9t5\" (UID: \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"
Nov 02 10:41:09.039366 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:09.039345    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmw2n\" (UniqueName: \"kubernetes.io/projected/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-kube-api-access-rmw2n\") pod \"maas-api-revoke-sub-85eb5cec-nm9t5\" (UID: \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"
Nov 02 10:41:09.117880 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:09.117835    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"
Nov 02 10:41:09.396053 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:09.396020    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"]
Nov 02 10:41:09.396791 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:41:09.396760    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ca18c1_a094_43a8_ba04_8a2e77e6fa21.slice/crio-25db29cc4354e2fdf3884dfe618936272d21d4059f93deb00c12654e15adbad9 WatchSource:0}: Error finding container 25db29cc4354e2fdf3884dfe618936272d21d4059f93deb00c12654e15adbad9: Status 404 returned error can't find the container with id 25db29cc4354e2fdf3884dfe618936272d21d4059f93deb00c12654e15adbad9
Nov 02 10:41:10.155595 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:10.155566    4449 generic.go:356] "Generic (PLEG): container finished" podID="e2ca18c1-a094-43a8-ba04-8a2e77e6fa21" containerID="97b89d82027ee3f15a82f9283195852eed367a862be4ccee47976a88ff2e2981" exitCode=0
Nov 02 10:41:10.155876 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:10.155654    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5" event={"ID":"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21","Type":"ContainerDied","Data":"97b89d82027ee3f15a82f9283195852eed367a862be4ccee47976a88ff2e2981"}
Nov 02 10:41:10.155876 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:10.155706    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5" event={"ID":"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21","Type":"ContainerStarted","Data":"25db29cc4354e2fdf3884dfe618936272d21d4059f93deb00c12654e15adbad9"}
Nov 02 10:41:11.296488 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:11.296449    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"
Nov 02 10:41:11.343436 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:11.343412    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-xks-ca\") pod \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\" (UID: \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\") "
Nov 02 10:41:11.343581 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:11.343498    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-kube-api-access-rmw2n\" (UniqueName: \"kubernetes.io/projected/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-kube-api-access-rmw2n\") pod \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\" (UID: \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\") "
Nov 02 10:41:11.343581 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:11.343525    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-openshift-ca\") pod \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\" (UID: \"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21\") "
Nov 02 10:41:11.343916 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:11.343891    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-openshift-ca" pod "e2ca18c1-a094-43a8-ba04-8a2e77e6fa21" (UID: "e2ca18c1-a094-43a8-ba04-8a2e77e6fa21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:41:11.345515 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:11.345493    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-xks-ca" pod "e2ca18c1-a094-43a8-ba04-8a2e77e6fa21" (UID: "e2ca18c1-a094-43a8-ba04-8a2e77e6fa21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:41:11.345656 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:11.345641    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-kube-api-access-rmw2n" pod "e2ca18c1-a094-43a8-ba04-8a2e77e6fa21" (UID: "e2ca18c1-a094-43a8-ba04-8a2e77e6fa21"). InnerVolumeSpecName "kube-api-access-rmw2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:41:11.445084 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:11.445032    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmw2n\" (UniqueName: \"kubernetes.io/projected/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-kube-api-access-rmw2n\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:11.445084 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:11.445054    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:11.445084 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:11.445063    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2ca18c1-a094-43a8-ba04-8a2e77e6fa21-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:12.167075 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:12.167045    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5" event={"ID":"e2ca18c1-a094-43a8-ba04-8a2e77e6fa21","Type":"ContainerDied","Data":"25db29cc4354e2fdf3884dfe618936272d21d4059f93deb00c12654e15adbad9"}
Nov 02 10:41:12.167206 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:12.167113    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25db29cc4354e2fdf3884dfe618936272d21d4059f93deb00c12654e15adbad9"
Nov 02 10:41:12.167206 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:12.167059    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85eb5cec-nm9t5"
Nov 02 10:41:19.011459 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.011428    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"]
Nov 02 10:41:19.011869 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.011857    4449 cpu_manager.go:414] "containerMap: removing container" podUID="e2ca18c1-a094-43a8-ba04-8a2e77e6fa21" containerName="revoke-keys"
Nov 02 10:41:19.011926 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.011870    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2ca18c1-a094-43a8-ba04-8a2e77e6fa21" containerName="revoke-keys"
Nov 02 10:41:19.011962 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.011945    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2ca18c1-a094-43a8-ba04-8a2e77e6fa21" containerName="revoke-keys"
Nov 02 10:41:19.015856 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.015839    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"
Nov 02 10:41:19.026231 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.020671    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:41:19.026231 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.021695    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:41:19.027371 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.027344    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"]
Nov 02 10:41:19.096813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.096785    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:41:19.096813 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.096806    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:41:19.107343 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.107320    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-102f4483-sqd7j\" (UID: \"77403013-b2ba-41f6-bdab-90b5fbbd681f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"
Nov 02 10:41:19.107455 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.107375    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-102f4483-sqd7j\" (UID: \"77403013-b2ba-41f6-bdab-90b5fbbd681f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"
Nov 02 10:41:19.107530 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.107508    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px8w2\" (UniqueName: \"kubernetes.io/projected/77403013-b2ba-41f6-bdab-90b5fbbd681f-kube-api-access-px8w2\") pod \"maas-api-revoke-sub-102f4483-sqd7j\" (UID: \"77403013-b2ba-41f6-bdab-90b5fbbd681f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"
Nov 02 10:41:19.208490 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.208436    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-102f4483-sqd7j\" (UID: \"77403013-b2ba-41f6-bdab-90b5fbbd681f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"
Nov 02 10:41:19.208783 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.208512    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-102f4483-sqd7j\" (UID: \"77403013-b2ba-41f6-bdab-90b5fbbd681f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"
Nov 02 10:41:19.208783 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.208572    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px8w2\" (UniqueName: \"kubernetes.io/projected/77403013-b2ba-41f6-bdab-90b5fbbd681f-kube-api-access-px8w2\") pod \"maas-api-revoke-sub-102f4483-sqd7j\" (UID: \"77403013-b2ba-41f6-bdab-90b5fbbd681f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"
Nov 02 10:41:19.209262 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.209237    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-102f4483-sqd7j\" (UID: \"77403013-b2ba-41f6-bdab-90b5fbbd681f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"
Nov 02 10:41:19.210901 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.210876    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-102f4483-sqd7j\" (UID: \"77403013-b2ba-41f6-bdab-90b5fbbd681f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"
Nov 02 10:41:19.220963 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.220927    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px8w2\" (UniqueName: \"kubernetes.io/projected/77403013-b2ba-41f6-bdab-90b5fbbd681f-kube-api-access-px8w2\") pod \"maas-api-revoke-sub-102f4483-sqd7j\" (UID: \"77403013-b2ba-41f6-bdab-90b5fbbd681f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"
Nov 02 10:41:19.332090 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.332071    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"
Nov 02 10:41:19.606910 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:19.606885    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"]
Nov 02 10:41:19.607710 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:41:19.607683    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77403013_b2ba_41f6_bdab_90b5fbbd681f.slice/crio-606157479f690a1dbde8529cf6d9ecbd40bbaa28fb0a1a38faabbb8cbcf57844 WatchSource:0}: Error finding container 606157479f690a1dbde8529cf6d9ecbd40bbaa28fb0a1a38faabbb8cbcf57844: Status 404 returned error can't find the container with id 606157479f690a1dbde8529cf6d9ecbd40bbaa28fb0a1a38faabbb8cbcf57844
Nov 02 10:41:20.216847 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:20.216802    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j" event={"ID":"77403013-b2ba-41f6-bdab-90b5fbbd681f","Type":"ContainerStarted","Data":"56fcfcfa34849f61819670b70b4b7a0c51f179239833b56e876b17571c7613e9"}
Nov 02 10:41:20.217359 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:20.216867    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j" event={"ID":"77403013-b2ba-41f6-bdab-90b5fbbd681f","Type":"ContainerStarted","Data":"606157479f690a1dbde8529cf6d9ecbd40bbaa28fb0a1a38faabbb8cbcf57844"}
Nov 02 10:41:20.237002 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:20.236951    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j" podStartSLOduration=2.236933504 podStartE2EDuration="2.236933504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:41:20.234733004 +0000 UTC m=+3957.173019801" watchObservedRunningTime="2026-11-02 10:41:20.236933504 +0000 UTC m=+3957.175220280"
Nov 02 10:41:21.227464 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:21.227427    4449 generic.go:356] "Generic (PLEG): container finished" podID="77403013-b2ba-41f6-bdab-90b5fbbd681f" containerID="56fcfcfa34849f61819670b70b4b7a0c51f179239833b56e876b17571c7613e9" exitCode=0
Nov 02 10:41:21.227961 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:21.227511    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j" event={"ID":"77403013-b2ba-41f6-bdab-90b5fbbd681f","Type":"ContainerDied","Data":"56fcfcfa34849f61819670b70b4b7a0c51f179239833b56e876b17571c7613e9"}
Nov 02 10:41:22.939138 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:22.938820    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"
Nov 02 10:41:22.959572 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:22.958770    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-xks-ca\") pod \"77403013-b2ba-41f6-bdab-90b5fbbd681f\" (UID: \"77403013-b2ba-41f6-bdab-90b5fbbd681f\") "
Nov 02 10:41:22.959572 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:22.958877    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-openshift-ca\") pod \"77403013-b2ba-41f6-bdab-90b5fbbd681f\" (UID: \"77403013-b2ba-41f6-bdab-90b5fbbd681f\") "
Nov 02 10:41:22.959572 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:22.958936    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77403013-b2ba-41f6-bdab-90b5fbbd681f-kube-api-access-px8w2\" (UniqueName: \"kubernetes.io/projected/77403013-b2ba-41f6-bdab-90b5fbbd681f-kube-api-access-px8w2\") pod \"77403013-b2ba-41f6-bdab-90b5fbbd681f\" (UID: \"77403013-b2ba-41f6-bdab-90b5fbbd681f\") "
Nov 02 10:41:22.960513 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:22.960472    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-openshift-ca" pod "77403013-b2ba-41f6-bdab-90b5fbbd681f" (UID: "77403013-b2ba-41f6-bdab-90b5fbbd681f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:41:22.993657 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:22.993609    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-xks-ca" pod "77403013-b2ba-41f6-bdab-90b5fbbd681f" (UID: "77403013-b2ba-41f6-bdab-90b5fbbd681f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:41:22.993937 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:22.993901    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77403013-b2ba-41f6-bdab-90b5fbbd681f-kube-api-access-px8w2" pod "77403013-b2ba-41f6-bdab-90b5fbbd681f" (UID: "77403013-b2ba-41f6-bdab-90b5fbbd681f"). InnerVolumeSpecName "kube-api-access-px8w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:41:23.060148 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:23.059884    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:23.060148 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:23.059917    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px8w2\" (UniqueName: \"kubernetes.io/projected/77403013-b2ba-41f6-bdab-90b5fbbd681f-kube-api-access-px8w2\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:23.060148 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:23.059933    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77403013-b2ba-41f6-bdab-90b5fbbd681f-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:23.315199 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:23.315165    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j" event={"ID":"77403013-b2ba-41f6-bdab-90b5fbbd681f","Type":"ContainerDied","Data":"606157479f690a1dbde8529cf6d9ecbd40bbaa28fb0a1a38faabbb8cbcf57844"}
Nov 02 10:41:23.338475 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:23.338452    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="606157479f690a1dbde8529cf6d9ecbd40bbaa28fb0a1a38faabbb8cbcf57844"
Nov 02 10:41:23.338587 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:23.329408    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102f4483-sqd7j"
Nov 02 10:41:27.150542 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:27.150514    4449 scope.go:122] "RemoveContainer" containerID="9a370065c505ffc8c1a1a87a9b48d4c0642dc1a78f28abfb7c0da34b64ff03e4"
Nov 02 10:41:29.010194 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:41:29.010167    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77403013_b2ba_41f6_bdab_90b5fbbd681f.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 10:41:29.311908 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.311881    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"]
Nov 02 10:41:29.312293 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.312278    4449 cpu_manager.go:414] "containerMap: removing container" podUID="77403013-b2ba-41f6-bdab-90b5fbbd681f" containerName="revoke-keys"
Nov 02 10:41:29.312293 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.312291    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77403013-b2ba-41f6-bdab-90b5fbbd681f" containerName="revoke-keys"
Nov 02 10:41:29.312451 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.312377    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="77403013-b2ba-41f6-bdab-90b5fbbd681f" containerName="revoke-keys"
Nov 02 10:41:29.315422 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.315405    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"
Nov 02 10:41:29.354059 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.354038    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:41:29.357668 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.357646    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:41:29.362397 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.362376    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"]
Nov 02 10:41:29.367940 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.367918    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:41:29.368088 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.368075    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:41:29.375603 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.375579    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e872ac-vjpd7\" (UID: \"2846fb17-db99-4ae6-b76c-f03fe906397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"
Nov 02 10:41:29.375710 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.375627    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48qdz\" (UniqueName: \"kubernetes.io/projected/2846fb17-db99-4ae6-b76c-f03fe906397f-kube-api-access-48qdz\") pod \"maas-api-revoke-sub-d3e872ac-vjpd7\" (UID: \"2846fb17-db99-4ae6-b76c-f03fe906397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"
Nov 02 10:41:29.375779 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.375761    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e872ac-vjpd7\" (UID: \"2846fb17-db99-4ae6-b76c-f03fe906397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"
Nov 02 10:41:29.476850 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.476827    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e872ac-vjpd7\" (UID: \"2846fb17-db99-4ae6-b76c-f03fe906397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"
Nov 02 10:41:29.476985 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.476874    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48qdz\" (UniqueName: \"kubernetes.io/projected/2846fb17-db99-4ae6-b76c-f03fe906397f-kube-api-access-48qdz\") pod \"maas-api-revoke-sub-d3e872ac-vjpd7\" (UID: \"2846fb17-db99-4ae6-b76c-f03fe906397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"
Nov 02 10:41:29.476985 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.476936    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e872ac-vjpd7\" (UID: \"2846fb17-db99-4ae6-b76c-f03fe906397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"
Nov 02 10:41:29.477498 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.477478    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e872ac-vjpd7\" (UID: \"2846fb17-db99-4ae6-b76c-f03fe906397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"
Nov 02 10:41:29.479297 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.479268    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e872ac-vjpd7\" (UID: \"2846fb17-db99-4ae6-b76c-f03fe906397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"
Nov 02 10:41:29.489630 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.489597    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48qdz\" (UniqueName: \"kubernetes.io/projected/2846fb17-db99-4ae6-b76c-f03fe906397f-kube-api-access-48qdz\") pod \"maas-api-revoke-sub-d3e872ac-vjpd7\" (UID: \"2846fb17-db99-4ae6-b76c-f03fe906397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"
Nov 02 10:41:29.628697 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.628647    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"
Nov 02 10:41:29.906844 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:41:29.906817    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2846fb17_db99_4ae6_b76c_f03fe906397f.slice/crio-e75799baa5c9d1bc8b8eb3c56f560e20374fb8d30aeacd8c80af6112dbdee9d1 WatchSource:0}: Error finding container e75799baa5c9d1bc8b8eb3c56f560e20374fb8d30aeacd8c80af6112dbdee9d1: Status 404 returned error can't find the container with id e75799baa5c9d1bc8b8eb3c56f560e20374fb8d30aeacd8c80af6112dbdee9d1
Nov 02 10:41:29.907010 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:29.906990    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"]
Nov 02 10:41:30.363769 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:30.362734    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7" event={"ID":"2846fb17-db99-4ae6-b76c-f03fe906397f","Type":"ContainerStarted","Data":"549a9d123578612edf10d0686a13a2f4bdee35c4ead837e8f21e9fee4f4d356e"}
Nov 02 10:41:30.363769 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:30.362804    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7" event={"ID":"2846fb17-db99-4ae6-b76c-f03fe906397f","Type":"ContainerStarted","Data":"e75799baa5c9d1bc8b8eb3c56f560e20374fb8d30aeacd8c80af6112dbdee9d1"}
Nov 02 10:41:30.386035 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:30.385975    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7" podStartSLOduration=1.385953558 podStartE2EDuration="1.385953558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:41:30.383003111 +0000 UTC m=+3967.321289910" watchObservedRunningTime="2026-11-02 10:41:30.385953558 +0000 UTC m=+3967.324240333"
Nov 02 10:41:31.374490 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:31.374447    4449 generic.go:356] "Generic (PLEG): container finished" podID="2846fb17-db99-4ae6-b76c-f03fe906397f" containerID="549a9d123578612edf10d0686a13a2f4bdee35c4ead837e8f21e9fee4f4d356e" exitCode=0
Nov 02 10:41:31.374977 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:31.374513    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7" event={"ID":"2846fb17-db99-4ae6-b76c-f03fe906397f","Type":"ContainerDied","Data":"549a9d123578612edf10d0686a13a2f4bdee35c4ead837e8f21e9fee4f4d356e"}
Nov 02 10:41:33.303672 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:33.303369    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"
Nov 02 10:41:33.457624 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:33.457588    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-openshift-ca\") pod \"2846fb17-db99-4ae6-b76c-f03fe906397f\" (UID: \"2846fb17-db99-4ae6-b76c-f03fe906397f\") "
Nov 02 10:41:33.471558 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:33.471405    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-openshift-ca" pod "2846fb17-db99-4ae6-b76c-f03fe906397f" (UID: "2846fb17-db99-4ae6-b76c-f03fe906397f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:41:33.568836 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:33.568538    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7" event={"ID":"2846fb17-db99-4ae6-b76c-f03fe906397f","Type":"ContainerDied","Data":"e75799baa5c9d1bc8b8eb3c56f560e20374fb8d30aeacd8c80af6112dbdee9d1"}
Nov 02 10:41:33.569126 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:33.569085    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e75799baa5c9d1bc8b8eb3c56f560e20374fb8d30aeacd8c80af6112dbdee9d1"
Nov 02 10:41:33.569325 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:33.568611    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e872ac-vjpd7"
Nov 02 10:41:33.569719 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:33.569701    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2846fb17-db99-4ae6-b76c-f03fe906397f-kube-api-access-48qdz\" (UniqueName: \"kubernetes.io/projected/2846fb17-db99-4ae6-b76c-f03fe906397f-kube-api-access-48qdz\") pod \"2846fb17-db99-4ae6-b76c-f03fe906397f\" (UID: \"2846fb17-db99-4ae6-b76c-f03fe906397f\") "
Nov 02 10:41:33.570672 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:33.570546    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-xks-ca\") pod \"2846fb17-db99-4ae6-b76c-f03fe906397f\" (UID: \"2846fb17-db99-4ae6-b76c-f03fe906397f\") "
Nov 02 10:41:33.571292 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:33.571271    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:33.622093 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:33.622023    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-xks-ca" pod "2846fb17-db99-4ae6-b76c-f03fe906397f" (UID: "2846fb17-db99-4ae6-b76c-f03fe906397f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:41:33.624400 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:33.624358    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2846fb17-db99-4ae6-b76c-f03fe906397f-kube-api-access-48qdz" pod "2846fb17-db99-4ae6-b76c-f03fe906397f" (UID: "2846fb17-db99-4ae6-b76c-f03fe906397f"). InnerVolumeSpecName "kube-api-access-48qdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:41:33.682945 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:33.682897    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48qdz\" (UniqueName: \"kubernetes.io/projected/2846fb17-db99-4ae6-b76c-f03fe906397f-kube-api-access-48qdz\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:33.683139 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:33.683120    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2846fb17-db99-4ae6-b76c-f03fe906397f-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:36.545510 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:41:36.545471    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:41:37.496049 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:41:37.496021    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77403013_b2ba_41f6_bdab_90b5fbbd681f.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 10:41:39.020760 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:41:39.020728    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77403013_b2ba_41f6_bdab_90b5fbbd681f.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 10:41:39.459452 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.459426    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"]
Nov 02 10:41:39.459840 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.459828    4449 cpu_manager.go:414] "containerMap: removing container" podUID="2846fb17-db99-4ae6-b76c-f03fe906397f" containerName="revoke-keys"
Nov 02 10:41:39.459896 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.459841    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2846fb17-db99-4ae6-b76c-f03fe906397f" containerName="revoke-keys"
Nov 02 10:41:39.459934 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.459919    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="2846fb17-db99-4ae6-b76c-f03fe906397f" containerName="revoke-keys"
Nov 02 10:41:39.463230 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.463190    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"
Nov 02 10:41:39.468138 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.468116    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:41:39.469248 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.469204    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:41:39.473715 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.473693    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"]
Nov 02 10:41:39.508978 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.508951    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:41:39.509097 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.508979    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:41:39.539138 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.539112    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgcf2\" (UniqueName: \"kubernetes.io/projected/ea80b484-8030-4d93-997f-d24719bace2e-kube-api-access-fgcf2\") pod \"maas-api-revoke-sub-ff2d344d-ptsjm\" (UID: \"ea80b484-8030-4d93-997f-d24719bace2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"
Nov 02 10:41:39.539266 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.539157    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff2d344d-ptsjm\" (UID: \"ea80b484-8030-4d93-997f-d24719bace2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"
Nov 02 10:41:39.539266 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.539238    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff2d344d-ptsjm\" (UID: \"ea80b484-8030-4d93-997f-d24719bace2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"
Nov 02 10:41:39.639777 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.639752    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgcf2\" (UniqueName: \"kubernetes.io/projected/ea80b484-8030-4d93-997f-d24719bace2e-kube-api-access-fgcf2\") pod \"maas-api-revoke-sub-ff2d344d-ptsjm\" (UID: \"ea80b484-8030-4d93-997f-d24719bace2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"
Nov 02 10:41:39.639898 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.639786    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff2d344d-ptsjm\" (UID: \"ea80b484-8030-4d93-997f-d24719bace2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"
Nov 02 10:41:39.639898 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.639815    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff2d344d-ptsjm\" (UID: \"ea80b484-8030-4d93-997f-d24719bace2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"
Nov 02 10:41:39.640437 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.640411    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff2d344d-ptsjm\" (UID: \"ea80b484-8030-4d93-997f-d24719bace2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"
Nov 02 10:41:39.642281 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.642265    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff2d344d-ptsjm\" (UID: \"ea80b484-8030-4d93-997f-d24719bace2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"
Nov 02 10:41:39.654713 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.654689    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgcf2\" (UniqueName: \"kubernetes.io/projected/ea80b484-8030-4d93-997f-d24719bace2e-kube-api-access-fgcf2\") pod \"maas-api-revoke-sub-ff2d344d-ptsjm\" (UID: \"ea80b484-8030-4d93-997f-d24719bace2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"
Nov 02 10:41:39.781661 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:39.781603    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"
Nov 02 10:41:40.061520 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:40.061492    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"]
Nov 02 10:41:40.062432 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:41:40.062404    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea80b484_8030_4d93_997f_d24719bace2e.slice/crio-dbc8dc99c64d35572625b90f229f9999568a5ce4b88f2fd63734495574c7fbc3 WatchSource:0}: Error finding container dbc8dc99c64d35572625b90f229f9999568a5ce4b88f2fd63734495574c7fbc3: Status 404 returned error can't find the container with id dbc8dc99c64d35572625b90f229f9999568a5ce4b88f2fd63734495574c7fbc3
Nov 02 10:41:40.617355 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:40.617254    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm" event={"ID":"ea80b484-8030-4d93-997f-d24719bace2e","Type":"ContainerStarted","Data":"acfc4afd3c39a449eb96461c93517cd376a92a144082fffe0eaeb85960e47f0e"}
Nov 02 10:41:40.617355 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:40.617323    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm" event={"ID":"ea80b484-8030-4d93-997f-d24719bace2e","Type":"ContainerStarted","Data":"dbc8dc99c64d35572625b90f229f9999568a5ce4b88f2fd63734495574c7fbc3"}
Nov 02 10:41:40.640392 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:40.640334    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm" podStartSLOduration=1.640316407 podStartE2EDuration="1.640316407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:41:40.635546751 +0000 UTC m=+3977.573833549" watchObservedRunningTime="2026-11-02 10:41:40.640316407 +0000 UTC m=+3977.578603180"
Nov 02 10:41:41.640240 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:41.639782    4449 generic.go:356] "Generic (PLEG): container finished" podID="ea80b484-8030-4d93-997f-d24719bace2e" containerID="acfc4afd3c39a449eb96461c93517cd376a92a144082fffe0eaeb85960e47f0e" exitCode=0
Nov 02 10:41:41.640240 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:41.640005    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm" event={"ID":"ea80b484-8030-4d93-997f-d24719bace2e","Type":"ContainerDied","Data":"acfc4afd3c39a449eb96461c93517cd376a92a144082fffe0eaeb85960e47f0e"}
Nov 02 10:41:43.353416 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.353387    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"
Nov 02 10:41:43.473249 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.472661    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-xks-ca\") pod \"ea80b484-8030-4d93-997f-d24719bace2e\" (UID: \"ea80b484-8030-4d93-997f-d24719bace2e\") "
Nov 02 10:41:43.473249 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.472770    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea80b484-8030-4d93-997f-d24719bace2e-kube-api-access-fgcf2\" (UniqueName: \"kubernetes.io/projected/ea80b484-8030-4d93-997f-d24719bace2e-kube-api-access-fgcf2\") pod \"ea80b484-8030-4d93-997f-d24719bace2e\" (UID: \"ea80b484-8030-4d93-997f-d24719bace2e\") "
Nov 02 10:41:43.473249 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.472863    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-openshift-ca\") pod \"ea80b484-8030-4d93-997f-d24719bace2e\" (UID: \"ea80b484-8030-4d93-997f-d24719bace2e\") "
Nov 02 10:41:43.494235 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.490427    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-openshift-ca" pod "ea80b484-8030-4d93-997f-d24719bace2e" (UID: "ea80b484-8030-4d93-997f-d24719bace2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:41:43.542481 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.526337    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-xks-ca" pod "ea80b484-8030-4d93-997f-d24719bace2e" (UID: "ea80b484-8030-4d93-997f-d24719bace2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:41:43.543409 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.543369    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea80b484-8030-4d93-997f-d24719bace2e-kube-api-access-fgcf2" pod "ea80b484-8030-4d93-997f-d24719bace2e" (UID: "ea80b484-8030-4d93-997f-d24719bace2e"). InnerVolumeSpecName "kube-api-access-fgcf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:41:43.582240 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.578424    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgcf2\" (UniqueName: \"kubernetes.io/projected/ea80b484-8030-4d93-997f-d24719bace2e-kube-api-access-fgcf2\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:43.582433 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.582411    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:43.586240 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.582545    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea80b484-8030-4d93-997f-d24719bace2e-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:43.804291 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.800916    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:41:43.804291 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.800989    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:41:43.850260 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.850207    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm" event={"ID":"ea80b484-8030-4d93-997f-d24719bace2e","Type":"ContainerDied","Data":"dbc8dc99c64d35572625b90f229f9999568a5ce4b88f2fd63734495574c7fbc3"}
Nov 02 10:41:43.850540 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.850526    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbc8dc99c64d35572625b90f229f9999568a5ce4b88f2fd63734495574c7fbc3"
Nov 02 10:41:43.850679 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:43.850669    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2d344d-ptsjm"
Nov 02 10:41:49.031700 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:41:49.031663    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77403013_b2ba_41f6_bdab_90b5fbbd681f.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 10:41:49.709869 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.709836    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"]
Nov 02 10:41:49.710266 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.710253    4449 cpu_manager.go:414] "containerMap: removing container" podUID="ea80b484-8030-4d93-997f-d24719bace2e" containerName="revoke-keys"
Nov 02 10:41:49.710318 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.710268    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea80b484-8030-4d93-997f-d24719bace2e" containerName="revoke-keys"
Nov 02 10:41:49.710358 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.710349    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea80b484-8030-4d93-997f-d24719bace2e" containerName="revoke-keys"
Nov 02 10:41:49.713663 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.713646    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"
Nov 02 10:41:49.722297 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.721367    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:41:49.722297 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.721465    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:41:49.724164 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.724139    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"]
Nov 02 10:41:49.732371 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.732349    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:41:49.732484 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.732371    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:41:49.773841 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.773817    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq7zt\" (UniqueName: \"kubernetes.io/projected/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-kube-api-access-qq7zt\") pod \"maas-api-revoke-sub-e970ac35-4lhf6\" (UID: \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"
Nov 02 10:41:49.773954 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.773866    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e970ac35-4lhf6\" (UID: \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"
Nov 02 10:41:49.773954 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.773898    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e970ac35-4lhf6\" (UID: \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"
Nov 02 10:41:49.874692 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.874668    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq7zt\" (UniqueName: \"kubernetes.io/projected/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-kube-api-access-qq7zt\") pod \"maas-api-revoke-sub-e970ac35-4lhf6\" (UID: \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"
Nov 02 10:41:49.874791 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.874717    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e970ac35-4lhf6\" (UID: \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"
Nov 02 10:41:49.874791 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.874759    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e970ac35-4lhf6\" (UID: \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"
Nov 02 10:41:49.875414 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.875392    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e970ac35-4lhf6\" (UID: \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"
Nov 02 10:41:49.877083 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.877062    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e970ac35-4lhf6\" (UID: \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"
Nov 02 10:41:49.887555 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:49.887534    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq7zt\" (UniqueName: \"kubernetes.io/projected/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-kube-api-access-qq7zt\") pod \"maas-api-revoke-sub-e970ac35-4lhf6\" (UID: \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"
Nov 02 10:41:50.029965 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:50.029912    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"
Nov 02 10:41:50.303925 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:50.303901    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"]
Nov 02 10:41:50.304737 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:41:50.304713    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0f3e93_a803_433a_97ab_ba3df8fc05ce.slice/crio-088d6fdf37cd73c9a9fb7e89f5d062897f3f6ae206211d7ab6b1c5c7df60b226 WatchSource:0}: Error finding container 088d6fdf37cd73c9a9fb7e89f5d062897f3f6ae206211d7ab6b1c5c7df60b226: Status 404 returned error can't find the container with id 088d6fdf37cd73c9a9fb7e89f5d062897f3f6ae206211d7ab6b1c5c7df60b226
Nov 02 10:41:50.912359 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:50.912259    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6" event={"ID":"5c0f3e93-a803-433a-97ab-ba3df8fc05ce","Type":"ContainerStarted","Data":"22dccd7358e08f2e02c5cd7b122ae7e2a53dc0884b205dca8fbfb114f0c03057"}
Nov 02 10:41:50.912612 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:50.912362    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6" event={"ID":"5c0f3e93-a803-433a-97ab-ba3df8fc05ce","Type":"ContainerStarted","Data":"088d6fdf37cd73c9a9fb7e89f5d062897f3f6ae206211d7ab6b1c5c7df60b226"}
Nov 02 10:41:50.937967 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:50.937912    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6" podStartSLOduration=1.9378954990000001 podStartE2EDuration="1.937895499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:41:50.932324236 +0000 UTC m=+3987.870611011" watchObservedRunningTime="2026-11-02 10:41:50.937895499 +0000 UTC m=+3987.876182273"
Nov 02 10:41:51.933707 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:51.933671    4449 generic.go:356] "Generic (PLEG): container finished" podID="5c0f3e93-a803-433a-97ab-ba3df8fc05ce" containerID="22dccd7358e08f2e02c5cd7b122ae7e2a53dc0884b205dca8fbfb114f0c03057" exitCode=0
Nov 02 10:41:51.935408 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:51.935374    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6" event={"ID":"5c0f3e93-a803-433a-97ab-ba3df8fc05ce","Type":"ContainerDied","Data":"22dccd7358e08f2e02c5cd7b122ae7e2a53dc0884b205dca8fbfb114f0c03057"}
Nov 02 10:41:52.449713 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:41:52.449664    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77403013_b2ba_41f6_bdab_90b5fbbd681f.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 10:41:53.862399 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:53.862370    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"
Nov 02 10:41:53.994111 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:53.994076    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-xks-ca\") pod \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\" (UID: \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\") "
Nov 02 10:41:53.994391 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:53.994369    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-openshift-ca\") pod \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\" (UID: \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\") "
Nov 02 10:41:53.994645 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:53.994622    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-kube-api-access-qq7zt\" (UniqueName: \"kubernetes.io/projected/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-kube-api-access-qq7zt\") pod \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\" (UID: \"5c0f3e93-a803-433a-97ab-ba3df8fc05ce\") "
Nov 02 10:41:54.073506 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:54.073006    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-xks-ca" pod "5c0f3e93-a803-433a-97ab-ba3df8fc05ce" (UID: "5c0f3e93-a803-433a-97ab-ba3df8fc05ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:41:54.073506 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:54.073208    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-openshift-ca" pod "5c0f3e93-a803-433a-97ab-ba3df8fc05ce" (UID: "5c0f3e93-a803-433a-97ab-ba3df8fc05ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:41:54.086821 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:54.086508    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6" event={"ID":"5c0f3e93-a803-433a-97ab-ba3df8fc05ce","Type":"ContainerDied","Data":"088d6fdf37cd73c9a9fb7e89f5d062897f3f6ae206211d7ab6b1c5c7df60b226"}
Nov 02 10:41:54.086821 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:54.086569    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="088d6fdf37cd73c9a9fb7e89f5d062897f3f6ae206211d7ab6b1c5c7df60b226"
Nov 02 10:41:54.086821 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:54.086636    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e970ac35-4lhf6"
Nov 02 10:41:54.096935 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:54.096880    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:54.096935 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:54.096910    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:54.109504 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:54.109479    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-kube-api-access-qq7zt" pod "5c0f3e93-a803-433a-97ab-ba3df8fc05ce" (UID: "5c0f3e93-a803-433a-97ab-ba3df8fc05ce"). InnerVolumeSpecName "kube-api-access-qq7zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:41:54.202237 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:54.198808    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq7zt\" (UniqueName: \"kubernetes.io/projected/5c0f3e93-a803-433a-97ab-ba3df8fc05ce-kube-api-access-qq7zt\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:41:59.085980 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:41:59.085944    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77403013_b2ba_41f6_bdab_90b5fbbd681f.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 10:41:59.940672 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:59.940638    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"]
Nov 02 10:41:59.941085 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:59.941072    4449 cpu_manager.go:414] "containerMap: removing container" podUID="5c0f3e93-a803-433a-97ab-ba3df8fc05ce" containerName="revoke-keys"
Nov 02 10:41:59.941147 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:59.941087    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0f3e93-a803-433a-97ab-ba3df8fc05ce" containerName="revoke-keys"
Nov 02 10:41:59.941187 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:59.941174    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0f3e93-a803-433a-97ab-ba3df8fc05ce" containerName="revoke-keys"
Nov 02 10:41:59.944490 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:59.944475    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"
Nov 02 10:41:59.949055 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:59.949033    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:41:59.949875 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:59.949783    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:41:59.955591 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:59.954161    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"]
Nov 02 10:41:59.976250 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:59.976229    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:41:59.976433 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:41:59.976270    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:42:00.049681 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:00.049656    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69xmm\" (UniqueName: \"kubernetes.io/projected/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-kube-api-access-69xmm\") pod \"maas-api-revoke-sub-48b495f7-6rsxp\" (UID: \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"
Nov 02 10:42:00.049802 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:00.049698    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b495f7-6rsxp\" (UID: \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"
Nov 02 10:42:00.049862 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:00.049815    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b495f7-6rsxp\" (UID: \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"
Nov 02 10:42:00.150882 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:00.150857    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69xmm\" (UniqueName: \"kubernetes.io/projected/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-kube-api-access-69xmm\") pod \"maas-api-revoke-sub-48b495f7-6rsxp\" (UID: \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"
Nov 02 10:42:00.151210 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:00.150898    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b495f7-6rsxp\" (UID: \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"
Nov 02 10:42:00.151210 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:00.150964    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b495f7-6rsxp\" (UID: \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"
Nov 02 10:42:00.151710 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:00.151646    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b495f7-6rsxp\" (UID: \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"
Nov 02 10:42:00.153442 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:00.153417    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b495f7-6rsxp\" (UID: \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"
Nov 02 10:42:00.163847 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:00.163811    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69xmm\" (UniqueName: \"kubernetes.io/projected/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-kube-api-access-69xmm\") pod \"maas-api-revoke-sub-48b495f7-6rsxp\" (UID: \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"
Nov 02 10:42:00.261101 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:00.261048    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"
Nov 02 10:42:00.537792 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:00.535586    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"]
Nov 02 10:42:00.539641 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:42:00.539608    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8d6efd_ce61_4b17_8f20_c3d16efc9c2d.slice/crio-8bdcf6c29e3833fab6fae60c29c6fffd9b44cc2f42beff40e06194842740e327 WatchSource:0}: Error finding container 8bdcf6c29e3833fab6fae60c29c6fffd9b44cc2f42beff40e06194842740e327: Status 404 returned error can't find the container with id 8bdcf6c29e3833fab6fae60c29c6fffd9b44cc2f42beff40e06194842740e327
Nov 02 10:42:01.130935 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:01.130891    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp" event={"ID":"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d","Type":"ContainerStarted","Data":"568966b8d98658272d29c32313dbfae656a913d4b5f81535315160bc80cb90f6"}
Nov 02 10:42:01.131149 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:01.130957    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp" event={"ID":"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d","Type":"ContainerStarted","Data":"8bdcf6c29e3833fab6fae60c29c6fffd9b44cc2f42beff40e06194842740e327"}
Nov 02 10:42:02.151797 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:02.151758    4449 generic.go:356] "Generic (PLEG): container finished" podID="dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d" containerID="568966b8d98658272d29c32313dbfae656a913d4b5f81535315160bc80cb90f6" exitCode=0
Nov 02 10:42:02.156914 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:02.156785    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp" event={"ID":"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d","Type":"ContainerDied","Data":"568966b8d98658272d29c32313dbfae656a913d4b5f81535315160bc80cb90f6"}
Nov 02 10:42:03.922585 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:03.922559    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"
Nov 02 10:42:04.083291 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:04.082505    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-openshift-ca\") pod \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\" (UID: \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\") "
Nov 02 10:42:04.083291 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:04.082640    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-xks-ca\") pod \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\" (UID: \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\") "
Nov 02 10:42:04.083291 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:04.082739    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-kube-api-access-69xmm\" (UniqueName: \"kubernetes.io/projected/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-kube-api-access-69xmm\") pod \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\" (UID: \"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d\") "
Nov 02 10:42:04.100536 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:04.100507    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-openshift-ca" pod "dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d" (UID: "dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:42:04.121237 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:04.115337    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-kube-api-access-69xmm" pod "dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d" (UID: "dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d"). InnerVolumeSpecName "kube-api-access-69xmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:42:04.137637 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:04.137610    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-xks-ca" pod "dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d" (UID: "dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:42:04.196075 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:04.195917    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:04.196075 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:04.195951    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69xmm\" (UniqueName: \"kubernetes.io/projected/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-kube-api-access-69xmm\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:04.196075 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:04.195967    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:04.202251 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:04.200864    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp" event={"ID":"dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d","Type":"ContainerDied","Data":"8bdcf6c29e3833fab6fae60c29c6fffd9b44cc2f42beff40e06194842740e327"}
Nov 02 10:42:04.202251 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:04.200912    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bdcf6c29e3833fab6fae60c29c6fffd9b44cc2f42beff40e06194842740e327"
Nov 02 10:42:04.202251 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:04.200977    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b495f7-6rsxp"
Nov 02 10:42:06.545140 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:42:06.545103    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:42:07.500059 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:42:07.500031    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77403013_b2ba_41f6_bdab_90b5fbbd681f.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 10:42:09.097364 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:42:09.097332    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77403013_b2ba_41f6_bdab_90b5fbbd681f.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 10:42:10.171439 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.171407    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"]
Nov 02 10:42:10.171830 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.171817    4449 cpu_manager.go:414] "containerMap: removing container" podUID="dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d" containerName="revoke-keys"
Nov 02 10:42:10.171879 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.171831    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d" containerName="revoke-keys"
Nov 02 10:42:10.171932 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.171915    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d" containerName="revoke-keys"
Nov 02 10:42:10.175361 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.175337    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"
Nov 02 10:42:10.180012 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.179989    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:42:10.181364 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.181338    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:42:10.185663 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.185639    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"]
Nov 02 10:42:10.202672 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.202634    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:42:10.202672 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.202634    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:42:10.244584 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.244556    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-494d14b4-ncpmx\" (UID: \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"
Nov 02 10:42:10.244707 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.244610    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjh7\" (UniqueName: \"kubernetes.io/projected/c72d9eaf-7e20-499e-bebd-b94cef78b69f-kube-api-access-8vjh7\") pod \"maas-api-revoke-sub-494d14b4-ncpmx\" (UID: \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"
Nov 02 10:42:10.244777 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.244723    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-494d14b4-ncpmx\" (UID: \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"
Nov 02 10:42:10.345920 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.345894    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-494d14b4-ncpmx\" (UID: \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"
Nov 02 10:42:10.346019 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.345957    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-494d14b4-ncpmx\" (UID: \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"
Nov 02 10:42:10.346019 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.345989    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjh7\" (UniqueName: \"kubernetes.io/projected/c72d9eaf-7e20-499e-bebd-b94cef78b69f-kube-api-access-8vjh7\") pod \"maas-api-revoke-sub-494d14b4-ncpmx\" (UID: \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"
Nov 02 10:42:10.346521 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.346503    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-494d14b4-ncpmx\" (UID: \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"
Nov 02 10:42:10.348245 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.348207    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-494d14b4-ncpmx\" (UID: \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"
Nov 02 10:42:10.358668 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.358647    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjh7\" (UniqueName: \"kubernetes.io/projected/c72d9eaf-7e20-499e-bebd-b94cef78b69f-kube-api-access-8vjh7\") pod \"maas-api-revoke-sub-494d14b4-ncpmx\" (UID: \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"
Nov 02 10:42:10.490680 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.490628    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"
Nov 02 10:42:10.764819 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:10.764791    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"]
Nov 02 10:42:10.765306 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:42:10.765282    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72d9eaf_7e20_499e_bebd_b94cef78b69f.slice/crio-e1db7e5056a9e4c7e83828d250d91d6c93e843eaf207dc648d5be213f2eae5ba WatchSource:0}: Error finding container e1db7e5056a9e4c7e83828d250d91d6c93e843eaf207dc648d5be213f2eae5ba: Status 404 returned error can't find the container with id e1db7e5056a9e4c7e83828d250d91d6c93e843eaf207dc648d5be213f2eae5ba
Nov 02 10:42:11.256724 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:11.256687    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx" event={"ID":"c72d9eaf-7e20-499e-bebd-b94cef78b69f","Type":"ContainerStarted","Data":"2067c49f4061eec51b423b031665d1326aac3660e0e817fb3902c4e041a8333b"}
Nov 02 10:42:11.257188 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:11.256752    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx" event={"ID":"c72d9eaf-7e20-499e-bebd-b94cef78b69f","Type":"ContainerStarted","Data":"e1db7e5056a9e4c7e83828d250d91d6c93e843eaf207dc648d5be213f2eae5ba"}
Nov 02 10:42:11.280999 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:11.280941    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx" podStartSLOduration=1.280921231 podStartE2EDuration="1.280921231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:42:11.276314974 +0000 UTC m=+4008.214601774" watchObservedRunningTime="2026-11-02 10:42:11.280921231 +0000 UTC m=+4008.219208006"
Nov 02 10:42:12.268352 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:12.268314    4449 generic.go:356] "Generic (PLEG): container finished" podID="c72d9eaf-7e20-499e-bebd-b94cef78b69f" containerID="2067c49f4061eec51b423b031665d1326aac3660e0e817fb3902c4e041a8333b" exitCode=0
Nov 02 10:42:12.268893 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:12.268400    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx" event={"ID":"c72d9eaf-7e20-499e-bebd-b94cef78b69f","Type":"ContainerDied","Data":"2067c49f4061eec51b423b031665d1326aac3660e0e817fb3902c4e041a8333b"}
Nov 02 10:42:13.800271 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:13.784305    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:42:13.800271 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:13.784370    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:42:14.117689 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:14.117387    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"
Nov 02 10:42:14.245376 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:14.241506    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-openshift-ca\") pod \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\" (UID: \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\") "
Nov 02 10:42:14.245376 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:14.245348    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-xks-ca\") pod \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\" (UID: \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\") "
Nov 02 10:42:14.245619 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:14.245434    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72d9eaf-7e20-499e-bebd-b94cef78b69f-kube-api-access-8vjh7\" (UniqueName: \"kubernetes.io/projected/c72d9eaf-7e20-499e-bebd-b94cef78b69f-kube-api-access-8vjh7\") pod \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\" (UID: \"c72d9eaf-7e20-499e-bebd-b94cef78b69f\") "
Nov 02 10:42:14.247014 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:14.246292    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-openshift-ca" pod "c72d9eaf-7e20-499e-bebd-b94cef78b69f" (UID: "c72d9eaf-7e20-499e-bebd-b94cef78b69f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:42:14.296237 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:14.278349    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72d9eaf-7e20-499e-bebd-b94cef78b69f-kube-api-access-8vjh7" pod "c72d9eaf-7e20-499e-bebd-b94cef78b69f" (UID: "c72d9eaf-7e20-499e-bebd-b94cef78b69f"). InnerVolumeSpecName "kube-api-access-8vjh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:42:14.300765 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:14.300736    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-xks-ca" pod "c72d9eaf-7e20-499e-bebd-b94cef78b69f" (UID: "c72d9eaf-7e20-499e-bebd-b94cef78b69f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:42:14.367444 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:14.350626    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:14.367662 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:14.367644    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjh7\" (UniqueName: \"kubernetes.io/projected/c72d9eaf-7e20-499e-bebd-b94cef78b69f-kube-api-access-8vjh7\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:14.367787 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:14.366304    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx"
Nov 02 10:42:14.376809 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:14.366203    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-494d14b4-ncpmx" event={"ID":"c72d9eaf-7e20-499e-bebd-b94cef78b69f","Type":"ContainerDied","Data":"e1db7e5056a9e4c7e83828d250d91d6c93e843eaf207dc648d5be213f2eae5ba"}
Nov 02 10:42:14.376809 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:14.376600    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1db7e5056a9e4c7e83828d250d91d6c93e843eaf207dc648d5be213f2eae5ba"
Nov 02 10:42:14.470987 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:14.470954    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72d9eaf-7e20-499e-bebd-b94cef78b69f-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:14.743811 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:42:14.743778    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77403013_b2ba_41f6_bdab_90b5fbbd681f.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 10:42:14.744053 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:42:14.743952    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77403013_b2ba_41f6_bdab_90b5fbbd681f.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 10:42:19.108553 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:42:19.108515    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77403013_b2ba_41f6_bdab_90b5fbbd681f.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 10:42:20.399300 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.399271    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"]
Nov 02 10:42:20.399768 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.399688    4449 cpu_manager.go:414] "containerMap: removing container" podUID="c72d9eaf-7e20-499e-bebd-b94cef78b69f" containerName="revoke-keys"
Nov 02 10:42:20.399768 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.399698    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72d9eaf-7e20-499e-bebd-b94cef78b69f" containerName="revoke-keys"
Nov 02 10:42:20.399873 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.399772    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72d9eaf-7e20-499e-bebd-b94cef78b69f" containerName="revoke-keys"
Nov 02 10:42:20.403082 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.403064    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"
Nov 02 10:42:20.407838 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.407814    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:42:20.407991 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.407846    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:42:20.408076 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.407895    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:42:20.408146 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.407906    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:42:20.411543 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.411518    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"]
Nov 02 10:42:20.539181 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.539153    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5669b9db-2mjlj\" (UID: \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"
Nov 02 10:42:20.539364 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.539203    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5669b9db-2mjlj\" (UID: \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"
Nov 02 10:42:20.539364 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.539244    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cwvn\" (UniqueName: \"kubernetes.io/projected/1d4ef418-5941-4f54-91f1-14e18f6f1b16-kube-api-access-5cwvn\") pod \"maas-api-revoke-sub-5669b9db-2mjlj\" (UID: \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"
Nov 02 10:42:20.640600 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.640574    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5669b9db-2mjlj\" (UID: \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"
Nov 02 10:42:20.640718 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.640621    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5669b9db-2mjlj\" (UID: \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"
Nov 02 10:42:20.640718 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.640645    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cwvn\" (UniqueName: \"kubernetes.io/projected/1d4ef418-5941-4f54-91f1-14e18f6f1b16-kube-api-access-5cwvn\") pod \"maas-api-revoke-sub-5669b9db-2mjlj\" (UID: \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"
Nov 02 10:42:20.641246 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.641194    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5669b9db-2mjlj\" (UID: \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"
Nov 02 10:42:20.642926 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.642899    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5669b9db-2mjlj\" (UID: \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"
Nov 02 10:42:20.653117 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.653065    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cwvn\" (UniqueName: \"kubernetes.io/projected/1d4ef418-5941-4f54-91f1-14e18f6f1b16-kube-api-access-5cwvn\") pod \"maas-api-revoke-sub-5669b9db-2mjlj\" (UID: \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"
Nov 02 10:42:20.719282 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.719262    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"
Nov 02 10:42:20.996403 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:20.996376    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"]
Nov 02 10:42:20.997170 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:42:20.997144    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4ef418_5941_4f54_91f1_14e18f6f1b16.slice/crio-cf3f3da1b14e56e55c9e95d1f8b08cb9a093c7d745fd152b9b2c7ef55b499d94 WatchSource:0}: Error finding container cf3f3da1b14e56e55c9e95d1f8b08cb9a093c7d745fd152b9b2c7ef55b499d94: Status 404 returned error can't find the container with id cf3f3da1b14e56e55c9e95d1f8b08cb9a093c7d745fd152b9b2c7ef55b499d94
Nov 02 10:42:21.411265 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:21.408834    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj" event={"ID":"1d4ef418-5941-4f54-91f1-14e18f6f1b16","Type":"ContainerStarted","Data":"7022a4c81f54cdb00aff73085d61c93549ba72d7c7544c560de7e9527bda204b"}
Nov 02 10:42:21.411265 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:21.408927    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj" event={"ID":"1d4ef418-5941-4f54-91f1-14e18f6f1b16","Type":"ContainerStarted","Data":"cf3f3da1b14e56e55c9e95d1f8b08cb9a093c7d745fd152b9b2c7ef55b499d94"}
Nov 02 10:42:21.435763 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:21.435697    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj" podStartSLOduration=1.4356778399999999 podStartE2EDuration="1.43567784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:42:21.428524496 +0000 UTC m=+4018.366811299" watchObservedRunningTime="2026-11-02 10:42:21.43567784 +0000 UTC m=+4018.373964614"
Nov 02 10:42:22.419636 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:22.419603    4449 generic.go:356] "Generic (PLEG): container finished" podID="1d4ef418-5941-4f54-91f1-14e18f6f1b16" containerID="7022a4c81f54cdb00aff73085d61c93549ba72d7c7544c560de7e9527bda204b" exitCode=0
Nov 02 10:42:22.420152 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:22.419689    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj" event={"ID":"1d4ef418-5941-4f54-91f1-14e18f6f1b16","Type":"ContainerDied","Data":"7022a4c81f54cdb00aff73085d61c93549ba72d7c7544c560de7e9527bda204b"}
Nov 02 10:42:22.519373 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:42:22.519338    4449 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77403013_b2ba_41f6_bdab_90b5fbbd681f.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 10:42:24.256242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:24.255559    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"
Nov 02 10:42:24.501875 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:24.501394    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-xks-ca\") pod \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\" (UID: \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\") "
Nov 02 10:42:24.501875 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:24.501611    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d4ef418-5941-4f54-91f1-14e18f6f1b16-kube-api-access-5cwvn\" (UniqueName: \"kubernetes.io/projected/1d4ef418-5941-4f54-91f1-14e18f6f1b16-kube-api-access-5cwvn\") pod \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\" (UID: \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\") "
Nov 02 10:42:24.501875 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:24.501688    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-openshift-ca\") pod \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\" (UID: \"1d4ef418-5941-4f54-91f1-14e18f6f1b16\") "
Nov 02 10:42:24.503087 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:24.503048    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-openshift-ca" pod "1d4ef418-5941-4f54-91f1-14e18f6f1b16" (UID: "1d4ef418-5941-4f54-91f1-14e18f6f1b16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:42:24.524748 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:24.524477    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj" event={"ID":"1d4ef418-5941-4f54-91f1-14e18f6f1b16","Type":"ContainerDied","Data":"cf3f3da1b14e56e55c9e95d1f8b08cb9a093c7d745fd152b9b2c7ef55b499d94"}
Nov 02 10:42:24.524748 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:24.524536    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf3f3da1b14e56e55c9e95d1f8b08cb9a093c7d745fd152b9b2c7ef55b499d94"
Nov 02 10:42:24.524748 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:24.524600    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5669b9db-2mjlj"
Nov 02 10:42:24.530606 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:24.530581    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-xks-ca" pod "1d4ef418-5941-4f54-91f1-14e18f6f1b16" (UID: "1d4ef418-5941-4f54-91f1-14e18f6f1b16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:42:24.530865 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:24.530844    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4ef418-5941-4f54-91f1-14e18f6f1b16-kube-api-access-5cwvn" pod "1d4ef418-5941-4f54-91f1-14e18f6f1b16" (UID: "1d4ef418-5941-4f54-91f1-14e18f6f1b16"). InnerVolumeSpecName "kube-api-access-5cwvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:42:24.607199 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:24.607166    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cwvn\" (UniqueName: \"kubernetes.io/projected/1d4ef418-5941-4f54-91f1-14e18f6f1b16-kube-api-access-5cwvn\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:24.607411 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:24.607392    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:24.607553 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:24.607537    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4ef418-5941-4f54-91f1-14e18f6f1b16-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:30.634525 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.634491    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"]
Nov 02 10:42:30.635109 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.635089    4449 cpu_manager.go:414] "containerMap: removing container" podUID="1d4ef418-5941-4f54-91f1-14e18f6f1b16" containerName="revoke-keys"
Nov 02 10:42:30.635206 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.635112    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d4ef418-5941-4f54-91f1-14e18f6f1b16" containerName="revoke-keys"
Nov 02 10:42:30.635310 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.635296    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d4ef418-5941-4f54-91f1-14e18f6f1b16" containerName="revoke-keys"
Nov 02 10:42:30.638382 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.638362    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"
Nov 02 10:42:30.642990 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.642968    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:42:30.643732 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.643707    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:42:30.647558 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.647541    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"]
Nov 02 10:42:30.664862 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.664834    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fd291e0-sj7w9\" (UID: \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"
Nov 02 10:42:30.665011 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.664890    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5cxb\" (UniqueName: \"kubernetes.io/projected/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-kube-api-access-k5cxb\") pod \"maas-api-revoke-sub-8fd291e0-sj7w9\" (UID: \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"
Nov 02 10:42:30.665084 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.665045    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fd291e0-sj7w9\" (UID: \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"
Nov 02 10:42:30.680913 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.680887    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:42:30.681022 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.680985    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:42:30.765597 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.765569    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fd291e0-sj7w9\" (UID: \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"
Nov 02 10:42:30.765720 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.765632    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fd291e0-sj7w9\" (UID: \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"
Nov 02 10:42:30.765720 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.765691    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5cxb\" (UniqueName: \"kubernetes.io/projected/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-kube-api-access-k5cxb\") pod \"maas-api-revoke-sub-8fd291e0-sj7w9\" (UID: \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"
Nov 02 10:42:30.766257 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.766173    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fd291e0-sj7w9\" (UID: \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"
Nov 02 10:42:30.767866 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.767845    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fd291e0-sj7w9\" (UID: \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"
Nov 02 10:42:30.777767 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.777744    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5cxb\" (UniqueName: \"kubernetes.io/projected/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-kube-api-access-k5cxb\") pod \"maas-api-revoke-sub-8fd291e0-sj7w9\" (UID: \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"
Nov 02 10:42:30.955315 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:30.955244    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"
Nov 02 10:42:31.229781 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:31.229752    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"]
Nov 02 10:42:31.230240 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:42:31.230187    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01a10d9_ff11_4a09_8d2e_039a511ee7c6.slice/crio-1972e7e625768fdd9462c1a2a3c03598adc051f1f74f88abc054bbfd92d2eaed WatchSource:0}: Error finding container 1972e7e625768fdd9462c1a2a3c03598adc051f1f74f88abc054bbfd92d2eaed: Status 404 returned error can't find the container with id 1972e7e625768fdd9462c1a2a3c03598adc051f1f74f88abc054bbfd92d2eaed
Nov 02 10:42:31.570858 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:31.570834    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9" event={"ID":"c01a10d9-ff11-4a09-8d2e-039a511ee7c6","Type":"ContainerStarted","Data":"f9726df32cd6ceea6894a96bd62316c339eeb607c1eb45a9bc0987da197e1d29"}
Nov 02 10:42:31.570964 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:31.570886    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9" event={"ID":"c01a10d9-ff11-4a09-8d2e-039a511ee7c6","Type":"ContainerStarted","Data":"1972e7e625768fdd9462c1a2a3c03598adc051f1f74f88abc054bbfd92d2eaed"}
Nov 02 10:42:31.589743 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:31.589703    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9" podStartSLOduration=1.589690756 podStartE2EDuration="1.589690756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:42:31.588324311 +0000 UTC m=+4028.526611105" watchObservedRunningTime="2026-11-02 10:42:31.589690756 +0000 UTC m=+4028.527977529"
Nov 02 10:42:32.583039 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:32.582999    4449 generic.go:356] "Generic (PLEG): container finished" podID="c01a10d9-ff11-4a09-8d2e-039a511ee7c6" containerID="f9726df32cd6ceea6894a96bd62316c339eeb607c1eb45a9bc0987da197e1d29" exitCode=0
Nov 02 10:42:32.583562 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:32.583094    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9" event={"ID":"c01a10d9-ff11-4a09-8d2e-039a511ee7c6","Type":"ContainerDied","Data":"f9726df32cd6ceea6894a96bd62316c339eeb607c1eb45a9bc0987da197e1d29"}
Nov 02 10:42:34.430248 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:34.427532    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"
Nov 02 10:42:34.536565 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:34.536530    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-openshift-ca\") pod \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\" (UID: \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\") "
Nov 02 10:42:34.536842 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:34.536820    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-xks-ca\") pod \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\" (UID: \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\") "
Nov 02 10:42:34.560378 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:34.560347    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-kube-api-access-k5cxb\" (UniqueName: \"kubernetes.io/projected/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-kube-api-access-k5cxb\") pod \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\" (UID: \"c01a10d9-ff11-4a09-8d2e-039a511ee7c6\") "
Nov 02 10:42:34.650238 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:34.648674    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-xks-ca" pod "c01a10d9-ff11-4a09-8d2e-039a511ee7c6" (UID: "c01a10d9-ff11-4a09-8d2e-039a511ee7c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:42:34.650238 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:34.649053    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-openshift-ca" pod "c01a10d9-ff11-4a09-8d2e-039a511ee7c6" (UID: "c01a10d9-ff11-4a09-8d2e-039a511ee7c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:42:34.654237 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:34.651680    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-kube-api-access-k5cxb" pod "c01a10d9-ff11-4a09-8d2e-039a511ee7c6" (UID: "c01a10d9-ff11-4a09-8d2e-039a511ee7c6"). InnerVolumeSpecName "kube-api-access-k5cxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:42:34.686242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:34.683427    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9" event={"ID":"c01a10d9-ff11-4a09-8d2e-039a511ee7c6","Type":"ContainerDied","Data":"1972e7e625768fdd9462c1a2a3c03598adc051f1f74f88abc054bbfd92d2eaed"}
Nov 02 10:42:34.686242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:34.683502    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1972e7e625768fdd9462c1a2a3c03598adc051f1f74f88abc054bbfd92d2eaed"
Nov 02 10:42:34.686242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:34.683579    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fd291e0-sj7w9"
Nov 02 10:42:34.687672 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:34.687648    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:34.687819 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:34.687804    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:34.696242 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:34.687900    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5cxb\" (UniqueName: \"kubernetes.io/projected/c01a10d9-ff11-4a09-8d2e-039a511ee7c6-kube-api-access-k5cxb\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:36.544249 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:42:36.544200    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:42:40.929952 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:40.929919    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"]
Nov 02 10:42:40.930388 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:40.930373    4449 cpu_manager.go:414] "containerMap: removing container" podUID="c01a10d9-ff11-4a09-8d2e-039a511ee7c6" containerName="revoke-keys"
Nov 02 10:42:40.930436 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:40.930390    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01a10d9-ff11-4a09-8d2e-039a511ee7c6" containerName="revoke-keys"
Nov 02 10:42:40.930493 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:40.930483    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01a10d9-ff11-4a09-8d2e-039a511ee7c6" containerName="revoke-keys"
Nov 02 10:42:40.933792 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:40.933766    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"
Nov 02 10:42:40.937764 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:40.937739    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 10:42:40.939411 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:40.938732    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 10:42:40.942730 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:40.942197    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"]
Nov 02 10:42:40.966297 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:40.966275    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp7pf\" (UniqueName: \"kubernetes.io/projected/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-kube-api-access-cp7pf\") pod \"maas-api-revoke-sub-7acce813-zdxd4\" (UID: \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"
Nov 02 10:42:40.966408 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:40.966305    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7acce813-zdxd4\" (UID: \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"
Nov 02 10:42:40.966584 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:40.966561    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7acce813-zdxd4\" (UID: \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"
Nov 02 10:42:41.014309 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:41.014288    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:42:41.014421 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:41.014337    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:42:41.068033 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:41.068011    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7acce813-zdxd4\" (UID: \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"
Nov 02 10:42:41.068150 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:41.068053    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp7pf\" (UniqueName: \"kubernetes.io/projected/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-kube-api-access-cp7pf\") pod \"maas-api-revoke-sub-7acce813-zdxd4\" (UID: \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"
Nov 02 10:42:41.068150 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:41.068070    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7acce813-zdxd4\" (UID: \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"
Nov 02 10:42:41.068642 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:41.068625    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7acce813-zdxd4\" (UID: \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"
Nov 02 10:42:41.070375 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:41.070358    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7acce813-zdxd4\" (UID: \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"
Nov 02 10:42:41.080185 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:41.080168    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp7pf\" (UniqueName: \"kubernetes.io/projected/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-kube-api-access-cp7pf\") pod \"maas-api-revoke-sub-7acce813-zdxd4\" (UID: \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"
Nov 02 10:42:41.248768 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:41.248720    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"
Nov 02 10:42:41.523568 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:41.523538    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"]
Nov 02 10:42:41.523839 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:42:41.523818    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ad8e22_3437_47d7_b3ab_0e9fe027aa3e.slice/crio-d68b52484d5fa93c01ddf7fb6718a9abc151dcc3788247fad2283c5954b9b169 WatchSource:0}: Error finding container d68b52484d5fa93c01ddf7fb6718a9abc151dcc3788247fad2283c5954b9b169: Status 404 returned error can't find the container with id d68b52484d5fa93c01ddf7fb6718a9abc151dcc3788247fad2283c5954b9b169
Nov 02 10:42:41.726040 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:41.726010    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4" event={"ID":"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e","Type":"ContainerStarted","Data":"d68b52484d5fa93c01ddf7fb6718a9abc151dcc3788247fad2283c5954b9b169"}
Nov 02 10:42:42.747843 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:42.747754    4449 generic.go:356] "Generic (PLEG): container finished" podID="64ad8e22-3437-47d7-b3ab-0e9fe027aa3e" containerID="9585376eec52c9c90503526f9e824dbbdd1cfdc3d33ad29a85f42ad254799e19" exitCode=0
Nov 02 10:42:42.748507 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:42.748478    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4" event={"ID":"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e","Type":"ContainerDied","Data":"9585376eec52c9c90503526f9e824dbbdd1cfdc3d33ad29a85f42ad254799e19"}
Nov 02 10:42:43.736346 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:43.736307    4449 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-qwf92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:42:43.736629 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:43.736600    4449 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-qwf92" podUID="e85060bf-426e-4a87-9e44-3f4577417345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:42:44.458237 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:44.457346    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"
Nov 02 10:42:44.589845 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:44.589817    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-openshift-ca\") pod \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\" (UID: \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\") "
Nov 02 10:42:44.590005 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:44.589927    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-kube-api-access-cp7pf\" (UniqueName: \"kubernetes.io/projected/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-kube-api-access-cp7pf\") pod \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\" (UID: \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\") "
Nov 02 10:42:44.590005 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:44.589946    4449 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-xks-ca\") pod \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\" (UID: \"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e\") "
Nov 02 10:42:44.590202 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:44.590167    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-openshift-ca" pod "64ad8e22-3437-47d7-b3ab-0e9fe027aa3e" (UID: "64ad8e22-3437-47d7-b3ab-0e9fe027aa3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:42:44.592057 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:44.592026    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-kube-api-access-cp7pf" pod "64ad8e22-3437-47d7-b3ab-0e9fe027aa3e" (UID: "64ad8e22-3437-47d7-b3ab-0e9fe027aa3e"). InnerVolumeSpecName "kube-api-access-cp7pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:42:44.592149 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:44.592085    4449 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-xks-ca" pod "64ad8e22-3437-47d7-b3ab-0e9fe027aa3e" (UID: "64ad8e22-3437-47d7-b3ab-0e9fe027aa3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:42:44.691452 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:44.691400    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-openshift-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:44.691452 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:44.691421    4449 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp7pf\" (UniqueName: \"kubernetes.io/projected/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-kube-api-access-cp7pf\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:44.691452 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:44.691431    4449 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ad8e22-3437-47d7-b3ab-0e9fe027aa3e-maas-api-xks-ca\") on node \"ip-10-0-128-217.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:42:44.793344 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:44.793322    4449 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4"
Nov 02 10:42:44.793435 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:44.793358    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7acce813-zdxd4" event={"ID":"64ad8e22-3437-47d7-b3ab-0e9fe027aa3e","Type":"ContainerDied","Data":"d68b52484d5fa93c01ddf7fb6718a9abc151dcc3788247fad2283c5954b9b169"}
Nov 02 10:42:44.793435 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:44.793422    4449 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d68b52484d5fa93c01ddf7fb6718a9abc151dcc3788247fad2283c5954b9b169"
Nov 02 10:42:59.924133 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:42:59.924098    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-795bcb67b4-xgtc6_f384c8c8-d8e1-447f-b53a-cf98f7103e28/authorino/0.log"
Nov 02 10:43:06.546493 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:43:06.546453    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:43:12.191732 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:12.191703    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cwjgx_1cc56635-aa7a-4702-8575-2c73011a580f/manager/0.log"
Nov 02 10:43:14.717434 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:14.717407    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0b064c2b-2kjgv_11a344ae-7697-4999-938b-776c67f7ed14/revoke-keys/0.log"
Nov 02 10:43:14.895610 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:14.895585    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0dc73283-blq7w_89dbb619-6a43-4d72-9181-513a5673bd67/revoke-keys/0.log"
Nov 02 10:43:15.083369 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:15.083345    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-102f4483-sqd7j_77403013-b2ba-41f6-bdab-90b5fbbd681f/revoke-keys/0.log"
Nov 02 10:43:15.264475 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:15.264440    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-183de081-rcwdv_a83b5799-8708-41ea-b6be-935bd32e8e2e/revoke-keys/0.log"
Nov 02 10:43:15.442508 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:15.442458    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2b225abb-cq4fc_f03598b8-71e2-49f4-a904-dcbd3afa85db/revoke-keys/0.log"
Nov 02 10:43:15.613287 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:15.613266    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2caa5c6d-8v5rh_c56e0d31-d49f-416f-9d10-5b3494cdd3da/revoke-keys/0.log"
Nov 02 10:43:15.798407 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:15.798382    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-48b495f7-6rsxp_dd8d6efd-ce61-4b17-8f20-c3d16efc9c2d/revoke-keys/0.log"
Nov 02 10:43:15.975912 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:15.975895    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-494d14b4-ncpmx_c72d9eaf-7e20-499e-bebd-b94cef78b69f/revoke-keys/0.log"
Nov 02 10:43:16.161318 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:16.161273    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5669b9db-2mjlj_1d4ef418-5941-4f54-91f1-14e18f6f1b16/revoke-keys/0.log"
Nov 02 10:43:16.331839 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:16.331820    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6785a9e6-pwb44_2aac4725-b768-4e0b-b4b3-fd7a0c5542d0/revoke-keys/0.log"
Nov 02 10:43:16.515846 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:16.515800    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7a0e87e7-wfzr2_e95f40d2-3c86-4c43-8c40-b02c90413a19/revoke-keys/0.log"
Nov 02 10:43:16.696398 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:16.696369    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7acce813-zdxd4_64ad8e22-3437-47d7-b3ab-0e9fe027aa3e/revoke-keys/0.log"
Nov 02 10:43:16.875175 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:16.875148    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-85eb5cec-nm9t5_e2ca18c1-a094-43a8-ba04-8a2e77e6fa21/revoke-keys/0.log"
Nov 02 10:43:17.056996 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:17.056971    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8fd291e0-sj7w9_c01a10d9-ff11-4a09-8d2e-039a511ee7c6/revoke-keys/0.log"
Nov 02 10:43:17.230702 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:17.230653    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d3e872ac-vjpd7_2846fb17-db99-4ae6-b76c-f03fe906397f/revoke-keys/0.log"
Nov 02 10:43:17.400948 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:17.400922    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e970ac35-4lhf6_5c0f3e93-a803-433a-97ab-ba3df8fc05ce/revoke-keys/0.log"
Nov 02 10:43:17.581336 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:17.581306    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fd04fbcf-g4jph_5e7e88d2-bcc5-4296-8707-4b7f7a86de96/revoke-keys/0.log"
Nov 02 10:43:17.768498 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:17.768475    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fe5e5cc3-85k7k_5a628840-a776-4725-86f5-c2e153808300/revoke-keys/0.log"
Nov 02 10:43:17.946705 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:17.946659    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ff2d344d-ptsjm_ea80b484-8030-4d93-997f-d24719bace2e/revoke-keys/0.log"
Nov 02 10:43:20.047149 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:20.047118    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-795bcb67b4-xgtc6_f384c8c8-d8e1-447f-b53a-cf98f7103e28/authorino/0.log"
Nov 02 10:43:20.262576 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:20.262548    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b66fk_36151d2a-760c-4b7c-bcee-daf2d6d3a562/manager/0.log"
Nov 02 10:43:20.455197 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:20.455138    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tjlwm_11792e8e-2931-4725-aece-39791e1c90bb/manager/0.log"
Nov 02 10:43:20.626762 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:20.626742    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-52xrj_7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708/kuadrant-console-plugin/0.log"
Nov 02 10:43:21.265328 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:21.265302    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2l8n5_f719b895-f939-4354-b42a-9c252a5f7712/manager/0.log"
Nov 02 10:43:22.200288 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:22.200264    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b6585knkfm_8edd99e2-b897-47c4-bdff-129cf994aad3/istio-proxy/0.log"
Nov 02 10:43:23.118639 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:23.118611    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-6cftx_604d7b4e-efdd-4979-b544-fd3d4d1953a3/istio-proxy/0.log"
Nov 02 10:43:23.293338 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:23.293317    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-9nd6j_51e601b5-d915-4326-b964-523a4f66788d/istio-proxy/0.log"
Nov 02 10:43:24.183437 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:24.183415    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-mqzmk_de093953-9e5f-4d01-9e8f-0404f29e0b82/istio-proxy/0.log"
Nov 02 10:43:24.358050 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:24.358026    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-qwf92_e85060bf-426e-4a87-9e44-3f4577417345/istio-proxy/0.log"
Nov 02 10:43:24.734827 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:24.734802    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz_ccf08bc0-8270-410d-a5d4-93be687e910a/istio-proxy/0.log"
Nov 02 10:43:25.081711 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:25.081687    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-75fb58b5b7-xbsfb_3b4a41e8-9183-4a0a-b088-4a292363401f/payload-processing/0.log"
Nov 02 10:43:26.423780 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:26.423757    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vzmpw_9c8f9db2-7c01-4c64-9424-bd7175d19fa2/main/0.log"
Nov 02 10:43:26.778042 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:26.777978    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-6wwp5_97fc91f5-ccd5-4f9d-8e45-592003d2ecd2/main/0.log"
Nov 02 10:43:27.127720 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:27.127698    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-npjcg_15cd868f-fb32-4dd9-bea2-e4e09323d130/main/0.log"
Nov 02 10:43:27.306673 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:27.306650    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-56c8w_e47c04fc-e082-4ab7-8813-af36a87969b8/main/0.log"
Nov 02 10:43:27.476257 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:27.476201    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-6hb6w_6e5b3557-3492-4152-9e0a-50291c7d1853/main/0.log"
Nov 02 10:43:36.025955 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:36.025881    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"]
Nov 02 10:43:36.029348 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:36.029322    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a0e87e7-wfzr2"]
Nov 02 10:43:36.545389 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:43:36.545358    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:43:37.205603 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:37.205571    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95f40d2-3c86-4c43-8c40-b02c90413a19" path="/var/lib/kubelet/pods/e95f40d2-3c86-4c43-8c40-b02c90413a19/volumes"
Nov 02 10:43:38.276707 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.276672    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zwdzc/must-gather-fm8bf"]
Nov 02 10:43:38.277201 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.277098    4449 cpu_manager.go:414] "containerMap: removing container" podUID="64ad8e22-3437-47d7-b3ab-0e9fe027aa3e" containerName="revoke-keys"
Nov 02 10:43:38.277201 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.277109    4449 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ad8e22-3437-47d7-b3ab-0e9fe027aa3e" containerName="revoke-keys"
Nov 02 10:43:38.277201 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.277185    4449 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ad8e22-3437-47d7-b3ab-0e9fe027aa3e" containerName="revoke-keys"
Nov 02 10:43:38.280646 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.280630    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zwdzc/must-gather-fm8bf"
Nov 02 10:43:38.289037 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.289014    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zwdzc\"/\"openshift-service-ca.crt\""
Nov 02 10:43:38.289211 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.289036    4449 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zwdzc\"/\"default-dockercfg-ktm6x\""
Nov 02 10:43:38.289211 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.289073    4449 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zwdzc\"/\"kube-root-ca.crt\""
Nov 02 10:43:38.293049 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.293027    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zwdzc/must-gather-fm8bf"]
Nov 02 10:43:38.456921 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.456896    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50b25512-6b78-488f-8081-852c5ffb3b41-must-gather-output\") pod \"must-gather-fm8bf\" (UID: \"50b25512-6b78-488f-8081-852c5ffb3b41\") " pod="openshift-must-gather-zwdzc/must-gather-fm8bf"
Nov 02 10:43:38.457037 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.456939    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqnvx\" (UniqueName: \"kubernetes.io/projected/50b25512-6b78-488f-8081-852c5ffb3b41-kube-api-access-hqnvx\") pod \"must-gather-fm8bf\" (UID: \"50b25512-6b78-488f-8081-852c5ffb3b41\") " pod="openshift-must-gather-zwdzc/must-gather-fm8bf"
Nov 02 10:43:38.558314 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.558292    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50b25512-6b78-488f-8081-852c5ffb3b41-must-gather-output\") pod \"must-gather-fm8bf\" (UID: \"50b25512-6b78-488f-8081-852c5ffb3b41\") " pod="openshift-must-gather-zwdzc/must-gather-fm8bf"
Nov 02 10:43:38.558421 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.558333    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqnvx\" (UniqueName: \"kubernetes.io/projected/50b25512-6b78-488f-8081-852c5ffb3b41-kube-api-access-hqnvx\") pod \"must-gather-fm8bf\" (UID: \"50b25512-6b78-488f-8081-852c5ffb3b41\") " pod="openshift-must-gather-zwdzc/must-gather-fm8bf"
Nov 02 10:43:38.558629 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.558609    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50b25512-6b78-488f-8081-852c5ffb3b41-must-gather-output\") pod \"must-gather-fm8bf\" (UID: \"50b25512-6b78-488f-8081-852c5ffb3b41\") " pod="openshift-must-gather-zwdzc/must-gather-fm8bf"
Nov 02 10:43:38.570917 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.570893    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqnvx\" (UniqueName: \"kubernetes.io/projected/50b25512-6b78-488f-8081-852c5ffb3b41-kube-api-access-hqnvx\") pod \"must-gather-fm8bf\" (UID: \"50b25512-6b78-488f-8081-852c5ffb3b41\") " pod="openshift-must-gather-zwdzc/must-gather-fm8bf"
Nov 02 10:43:38.595814 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.595787    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zwdzc/must-gather-fm8bf"
Nov 02 10:43:38.923072 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.923047    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zwdzc/must-gather-fm8bf"]
Nov 02 10:43:38.923238 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:43:38.923203    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50b25512_6b78_488f_8081_852c5ffb3b41.slice/crio-21372d2ef4533c574c4b51e8aa26bd0125cc7ed7a821bf0c4aeed5aa7e72e13a WatchSource:0}: Error finding container 21372d2ef4533c574c4b51e8aa26bd0125cc7ed7a821bf0c4aeed5aa7e72e13a: Status 404 returned error can't find the container with id 21372d2ef4533c574c4b51e8aa26bd0125cc7ed7a821bf0c4aeed5aa7e72e13a
Nov 02 10:43:38.925138 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:38.925123    4449 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 10:43:39.198542 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:39.198485    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwdzc/must-gather-fm8bf" event={"ID":"50b25512-6b78-488f-8081-852c5ffb3b41","Type":"ContainerStarted","Data":"21372d2ef4533c574c4b51e8aa26bd0125cc7ed7a821bf0c4aeed5aa7e72e13a"}
Nov 02 10:43:40.207527 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:40.207490    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwdzc/must-gather-fm8bf" event={"ID":"50b25512-6b78-488f-8081-852c5ffb3b41","Type":"ContainerStarted","Data":"ce56bb9513916a43221d3ccf1ac6f5cfd3a406e99c457eac8ce39bb9c9f8a881"}
Nov 02 10:43:40.207966 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:40.207536    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwdzc/must-gather-fm8bf" event={"ID":"50b25512-6b78-488f-8081-852c5ffb3b41","Type":"ContainerStarted","Data":"a35b521529840b25ced67afecbd002d32c0e689ba9056d581ec618afbeade743"}
Nov 02 10:43:40.231243 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:40.229357    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zwdzc/must-gather-fm8bf" podStartSLOduration=1.410987493 podStartE2EDuration="2.229339548s" totalImagesPullingTime="818.352055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:43:40.225126763 +0000 UTC m=+4097.163413537" watchObservedRunningTime="2026-11-02 10:43:40.229339548 +0000 UTC m=+4097.167626324"
Nov 02 10:43:41.650212 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:41.650183    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8w4cb_69f3517a-6eb0-4b5a-a5de-fa56577626be/global-pull-secret-syncer/0.log"
Nov 02 10:43:41.992675 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:41.992437    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h48j7_ef320c69-127d-437a-87ed-cfcd9db66325/konnectivity-agent/0.log"
Nov 02 10:43:42.066138 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:42.066075    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-217.us-east-2.compute.internal_89488fe67d46e24a4bfb2dbd39ca56e0/haproxy/0.log"
Nov 02 10:43:45.578239 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:45.578149    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cwjgx_1cc56635-aa7a-4702-8575-2c73011a580f/manager/0.log"
Nov 02 10:43:47.031238 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:47.028158    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"]
Nov 02 10:43:47.063060 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:47.060754    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc73283-blq7w"]
Nov 02 10:43:47.211709 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:47.211677    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89dbb619-6a43-4d72-9181-513a5673bd67" path="/var/lib/kubelet/pods/89dbb619-6a43-4d72-9181-513a5673bd67/volumes"
Nov 02 10:43:47.639047 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:47.639005    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cwjgx_1cc56635-aa7a-4702-8575-2c73011a580f/manager/0.log"
Nov 02 10:43:49.385956 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:49.385816    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-795bcb67b4-xgtc6_f384c8c8-d8e1-447f-b53a-cf98f7103e28/authorino/0.log"
Nov 02 10:43:49.422647 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:49.422616    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b66fk_36151d2a-760c-4b7c-bcee-daf2d6d3a562/manager/0.log"
Nov 02 10:43:49.474133 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:49.474096    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tjlwm_11792e8e-2931-4725-aece-39791e1c90bb/manager/0.log"
Nov 02 10:43:49.510077 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:49.510048    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-52xrj_7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708/kuadrant-console-plugin/0.log"
Nov 02 10:43:49.740341 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:49.740261    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2l8n5_f719b895-f939-4354-b42a-9c252a5f7712/manager/0.log"
Nov 02 10:43:49.980469 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:49.980419    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cwjgx_1cc56635-aa7a-4702-8575-2c73011a580f/manager/0.log"
Nov 02 10:43:51.355534 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:51.355500    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d60d3785-1f6c-405a-8785-048827802da2/alertmanager/0.log"
Nov 02 10:43:51.392442 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:51.392405    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d60d3785-1f6c-405a-8785-048827802da2/config-reloader/0.log"
Nov 02 10:43:51.417151 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:51.417125    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d60d3785-1f6c-405a-8785-048827802da2/kube-rbac-proxy-web/0.log"
Nov 02 10:43:51.443541 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:51.443515    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d60d3785-1f6c-405a-8785-048827802da2/kube-rbac-proxy/0.log"
Nov 02 10:43:51.471311 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:51.471286    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d60d3785-1f6c-405a-8785-048827802da2/kube-rbac-proxy-metric/0.log"
Nov 02 10:43:51.503529 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:51.503502    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d60d3785-1f6c-405a-8785-048827802da2/prom-label-proxy/0.log"
Nov 02 10:43:51.551105 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:51.551079    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d60d3785-1f6c-405a-8785-048827802da2/init-config-reloader/0.log"
Nov 02 10:43:51.895299 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:51.895272    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8cgjf_4dce88eb-70b7-4d91-bd14-533e9c3c18cf/node-exporter/0.log"
Nov 02 10:43:51.921331 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:51.921307    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8cgjf_4dce88eb-70b7-4d91-bd14-533e9c3c18cf/kube-rbac-proxy/0.log"
Nov 02 10:43:51.947894 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:51.947870    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8cgjf_4dce88eb-70b7-4d91-bd14-533e9c3c18cf/init-textfile/0.log"
Nov 02 10:43:52.075393 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.075360    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-fvz6m_e7929f42-1f6b-4ad9-866b-da15457ab4ec/kube-rbac-proxy-main/0.log"
Nov 02 10:43:52.103119 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.103094    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-fvz6m_e7929f42-1f6b-4ad9-866b-da15457ab4ec/kube-rbac-proxy-self/0.log"
Nov 02 10:43:52.131109 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.131081    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-fvz6m_e7929f42-1f6b-4ad9-866b-da15457ab4ec/openshift-state-metrics/0.log"
Nov 02 10:43:52.234283 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.233669    4449 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"]
Nov 02 10:43:52.235899 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.235168    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b6585knkfm_8edd99e2-b897-47c4-bdff-129cf994aad3/istio-proxy/0.log"
Nov 02 10:43:52.239525 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.239484    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.246004 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.245976    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"]
Nov 02 10:43:52.356080 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.356050    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv9sp\" (UniqueName: \"kubernetes.io/projected/12030bab-186b-476b-a433-814ff0dd34e5-kube-api-access-lv9sp\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.356653 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.356099    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12030bab-186b-476b-a433-814ff0dd34e5-podres\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.356653 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.356206    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12030bab-186b-476b-a433-814ff0dd34e5-proc\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.356653 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.356389    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12030bab-186b-476b-a433-814ff0dd34e5-sys\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.356653 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.356517    4449 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12030bab-186b-476b-a433-814ff0dd34e5-lib-modules\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.458776 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.458702    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12030bab-186b-476b-a433-814ff0dd34e5-podres\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.458776 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.458771    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12030bab-186b-476b-a433-814ff0dd34e5-proc\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.459072 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.458850    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12030bab-186b-476b-a433-814ff0dd34e5-sys\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.459072 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.458853    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12030bab-186b-476b-a433-814ff0dd34e5-podres\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.459072 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.458878    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12030bab-186b-476b-a433-814ff0dd34e5-proc\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.459072 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.458925    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12030bab-186b-476b-a433-814ff0dd34e5-sys\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.459072 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.458929    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12030bab-186b-476b-a433-814ff0dd34e5-lib-modules\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.459072 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.459003    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12030bab-186b-476b-a433-814ff0dd34e5-lib-modules\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.459072 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.459039    4449 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv9sp\" (UniqueName: \"kubernetes.io/projected/12030bab-186b-476b-a433-814ff0dd34e5-kube-api-access-lv9sp\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.475237 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.473765    4449 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv9sp\" (UniqueName: \"kubernetes.io/projected/12030bab-186b-476b-a433-814ff0dd34e5-kube-api-access-lv9sp\") pod \"perf-node-gather-daemonset-htfgk\" (UID: \"12030bab-186b-476b-a433-814ff0dd34e5\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.492362 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.492287    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7df55dc5f-g8p4f_d915c045-08fd-4ca6-a35d-0fa360cee77c/telemeter-client/0.log"
Nov 02 10:43:52.518511 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.518482    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7df55dc5f-g8p4f_d915c045-08fd-4ca6-a35d-0fa360cee77c/reload/0.log"
Nov 02 10:43:52.544280 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.544250    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7df55dc5f-g8p4f_d915c045-08fd-4ca6-a35d-0fa360cee77c/kube-rbac-proxy/0.log"
Nov 02 10:43:52.561559 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.561509    4449 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:52.586558 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.586533    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d9cc5b7-z88nh_5f39265c-2507-4c75-969c-33cdaea8f677/thanos-query/0.log"
Nov 02 10:43:52.622920 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.620130    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d9cc5b7-z88nh_5f39265c-2507-4c75-969c-33cdaea8f677/kube-rbac-proxy-web/0.log"
Nov 02 10:43:52.653675 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.652686    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d9cc5b7-z88nh_5f39265c-2507-4c75-969c-33cdaea8f677/kube-rbac-proxy/0.log"
Nov 02 10:43:52.688961 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.688933    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d9cc5b7-z88nh_5f39265c-2507-4c75-969c-33cdaea8f677/prom-label-proxy/0.log"
Nov 02 10:43:52.720447 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.720414    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d9cc5b7-z88nh_5f39265c-2507-4c75-969c-33cdaea8f677/kube-rbac-proxy-rules/0.log"
Nov 02 10:43:52.746734 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.746671    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d9cc5b7-z88nh_5f39265c-2507-4c75-969c-33cdaea8f677/kube-rbac-proxy-metrics/0.log"
Nov 02 10:43:52.901747 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:52.901703    4449 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"]
Nov 02 10:43:52.909261 ip-10-0-128-217 kubenswrapper[4449]: W1102 10:43:52.907077    4449 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod12030bab_186b_476b_a433_814ff0dd34e5.slice/crio-ef624817f4b6b79512b37d801509eff93d59b33f7256d571d5375d4ea449d32b WatchSource:0}: Error finding container ef624817f4b6b79512b37d801509eff93d59b33f7256d571d5375d4ea449d32b: Status 404 returned error can't find the container with id ef624817f4b6b79512b37d801509eff93d59b33f7256d571d5375d4ea449d32b
Nov 02 10:43:53.164403 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:53.164353    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-6cftx_604d7b4e-efdd-4979-b544-fd3d4d1953a3/istio-proxy/0.log"
Nov 02 10:43:53.190680 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:53.190651    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-9nd6j_51e601b5-d915-4326-b964-523a4f66788d/istio-proxy/0.log"
Nov 02 10:43:53.282963 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:53.282930    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-mqzmk_de093953-9e5f-4d01-9e8f-0404f29e0b82/istio-proxy/0.log"
Nov 02 10:43:53.304684 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:53.304659    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-qwf92_e85060bf-426e-4a87-9e44-3f4577417345/istio-proxy/0.log"
Nov 02 10:43:53.348898 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:53.344859    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz_ccf08bc0-8270-410d-a5d4-93be687e910a/istio-proxy/0.log"
Nov 02 10:43:53.374977 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:53.374948    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-75fb58b5b7-xbsfb_3b4a41e8-9183-4a0a-b088-4a292363401f/payload-processing/0.log"
Nov 02 10:43:53.428439 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:53.428380    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk" event={"ID":"12030bab-186b-476b-a433-814ff0dd34e5","Type":"ContainerStarted","Data":"c2b075836250d10c78a2993af5a4d2aebb599a5b99a847e6fd69474690c40d6d"}
Nov 02 10:43:53.428667 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:53.428530    4449 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk" event={"ID":"12030bab-186b-476b-a433-814ff0dd34e5","Type":"ContainerStarted","Data":"ef624817f4b6b79512b37d801509eff93d59b33f7256d571d5375d4ea449d32b"}
Nov 02 10:43:53.428817 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:53.428795    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:43:53.448756 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:53.448681    4449 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk" podStartSLOduration=1.448661971 podStartE2EDuration="1.448661971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 10:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:43:53.447384299 +0000 UTC m=+4110.385671073" watchObservedRunningTime="2026-11-02 10:43:53.448661971 +0000 UTC m=+4110.386948746"
Nov 02 10:43:55.029587 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:55.029495    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58bc4dbc95-6p2qz_accacbf0-bbb4-4ffb-9ebc-dcd26fc62a37/console/0.log"
Nov 02 10:43:55.074471 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:55.074447    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-g6wb2_b30515be-06ba-4ccf-95fe-b449af52203c/download-server/0.log"
Nov 02 10:43:56.522405 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:56.522380    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-795bcb67b4-xgtc6_f384c8c8-d8e1-447f-b53a-cf98f7103e28/authorino/0.log"
Nov 02 10:43:56.542230 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:56.542190    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b66fk_36151d2a-760c-4b7c-bcee-daf2d6d3a562/manager/0.log"
Nov 02 10:43:56.559267 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:56.559239    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tjlwm_11792e8e-2931-4725-aece-39791e1c90bb/manager/0.log"
Nov 02 10:43:56.577859 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:56.577838    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-52xrj_7df6b4fa-47c6-4bd7-91fa-0dd2ad23a708/kuadrant-console-plugin/0.log"
Nov 02 10:43:56.747051 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:56.747012    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-2l8n5_f719b895-f939-4354-b42a-9c252a5f7712/manager/0.log"
Nov 02 10:43:57.123479 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:57.123455    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sjj6v_60edc0c2-d978-40b5-b155-152019a3bdf5/dns/0.log"
Nov 02 10:43:57.150863 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:57.150837    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sjj6v_60edc0c2-d978-40b5-b155-152019a3bdf5/kube-rbac-proxy/0.log"
Nov 02 10:43:57.270761 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:57.270734    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r85rg_8e9ee3cd-e542-40a1-a18f-2843dcd82a69/dns-node-resolver/0.log"
Nov 02 10:43:57.882862 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:57.882833    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84b96dbbff-vb64t_67fc47bc-862f-4311-918f-2af036ebd4b4/registry/0.log"
Nov 02 10:43:57.937915 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:57.937886    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hxwln_0556aaf2-dc7e-4e57-a721-4e0ee6cfd923/node-ca/0.log"
Nov 02 10:43:58.986204 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:58.986179    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b6585knkfm_8edd99e2-b897-47c4-bdff-129cf994aad3/istio-proxy/0.log"
Nov 02 10:43:59.453728 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:43:59.453699    4449 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-htfgk"
Nov 02 10:44:00.232246 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:00.231966    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-6cftx_604d7b4e-efdd-4979-b544-fd3d4d1953a3/istio-proxy/0.log"
Nov 02 10:44:00.262766 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:00.262719    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-9nd6j_51e601b5-d915-4326-b964-523a4f66788d/istio-proxy/0.log"
Nov 02 10:44:00.427441 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:00.427406    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-mqzmk_de093953-9e5f-4d01-9e8f-0404f29e0b82/istio-proxy/0.log"
Nov 02 10:44:00.462151 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:00.462112    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-qwf92_e85060bf-426e-4a87-9e44-3f4577417345/istio-proxy/0.log"
Nov 02 10:44:00.526497 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:00.526423    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz_ccf08bc0-8270-410d-a5d4-93be687e910a/istio-proxy/0.log"
Nov 02 10:44:00.593094 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:00.593065    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-75fb58b5b7-xbsfb_3b4a41e8-9183-4a0a-b088-4a292363401f/payload-processing/0.log"
Nov 02 10:44:02.645381 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:02.645354    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b6585knkfm_8edd99e2-b897-47c4-bdff-129cf994aad3/istio-proxy/0.log"
Nov 02 10:44:02.647209 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:02.647188    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l7v8r_b0a20d22-7ea2-4cb8-a46f-d48043889d6f/serve-healthcheck-canary/0.log"
Nov 02 10:44:03.547994 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:03.547925    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-smlj2_c108bc48-9b97-46d1-a353-04a6dbc9ea83/kube-rbac-proxy/0.log"
Nov 02 10:44:03.553640 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:03.553609    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-6cftx_604d7b4e-efdd-4979-b544-fd3d4d1953a3/istio-proxy/0.log"
Nov 02 10:44:03.575550 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:03.575508    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-9nd6j_51e601b5-d915-4326-b964-523a4f66788d/istio-proxy/0.log"
Nov 02 10:44:03.576166 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:03.576146    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-smlj2_c108bc48-9b97-46d1-a353-04a6dbc9ea83/exporter/0.log"
Nov 02 10:44:03.608743 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:03.608718    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-smlj2_c108bc48-9b97-46d1-a353-04a6dbc9ea83/extractor/0.log"
Nov 02 10:44:03.685027 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:03.684985    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-mqzmk_de093953-9e5f-4d01-9e8f-0404f29e0b82/istio-proxy/0.log"
Nov 02 10:44:03.714453 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:03.714426    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-qwf92_e85060bf-426e-4a87-9e44-3f4577417345/istio-proxy/0.log"
Nov 02 10:44:03.762046 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:03.762002    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz_ccf08bc0-8270-410d-a5d4-93be687e910a/istio-proxy/0.log"
Nov 02 10:44:03.795928 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:03.795893    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-75fb58b5b7-xbsfb_3b4a41e8-9183-4a0a-b088-4a292363401f/payload-processing/0.log"
Nov 02 10:44:04.667628 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:04.667602    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-f7698d755-kkqg6_14ae7976-d81d-4847-97fe-ebe0cac032ec/manager/0.log"
Nov 02 10:44:04.681906 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:04.681880    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-f7698d755-kkqg6_14ae7976-d81d-4847-97fe-ebe0cac032ec/copy-manifests/0.log"
Nov 02 10:44:05.257875 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:05.257841    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cwjgx_1cc56635-aa7a-4702-8575-2c73011a580f/manager/0.log"
Nov 02 10:44:06.420816 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:06.420783    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b6585knkfm_8edd99e2-b897-47c4-bdff-129cf994aad3/istio-proxy/0.log"
Nov 02 10:44:06.464983 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:06.464951    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-f7698d755-kkqg6_14ae7976-d81d-4847-97fe-ebe0cac032ec/manager/0.log"
Nov 02 10:44:06.490884 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:06.490858    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-f7698d755-kkqg6_14ae7976-d81d-4847-97fe-ebe0cac032ec/copy-manifests/0.log"
Nov 02 10:44:06.564518 ip-10-0-128-217 kubenswrapper[4449]: E1102 10:44:06.564483    4449 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smlj2" containerName="extractor"
Nov 02 10:44:07.157110 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:07.157083    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b6585knkfm_8edd99e2-b897-47c4-bdff-129cf994aad3/istio-proxy/0.log"
Nov 02 10:44:07.348579 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:07.348548    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-6cftx_604d7b4e-efdd-4979-b544-fd3d4d1953a3/istio-proxy/0.log"
Nov 02 10:44:07.372032 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:07.372000    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-9nd6j_51e601b5-d915-4326-b964-523a4f66788d/istio-proxy/0.log"
Nov 02 10:44:07.480304 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:07.480231    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-mqzmk_de093953-9e5f-4d01-9e8f-0404f29e0b82/istio-proxy/0.log"
Nov 02 10:44:07.507004 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:07.506981    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-qwf92_e85060bf-426e-4a87-9e44-3f4577417345/istio-proxy/0.log"
Nov 02 10:44:07.542513 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:07.542485    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz_ccf08bc0-8270-410d-a5d4-93be687e910a/istio-proxy/0.log"
Nov 02 10:44:07.588729 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:07.588701    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-75fb58b5b7-xbsfb_3b4a41e8-9183-4a0a-b088-4a292363401f/payload-processing/0.log"
Nov 02 10:44:07.969484 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:07.969451    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-7cf664dd55-w9468_00095481-c8a0-4520-984a-3f426e49a758/manager/0.log"
Nov 02 10:44:08.035644 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:08.035362    4449 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"]
Nov 02 10:44:08.040633 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:08.040604    4449 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b064c2b-2kjgv"]
Nov 02 10:44:08.086978 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:08.086949    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-6cftx_604d7b4e-efdd-4979-b544-fd3d4d1953a3/istio-proxy/0.log"
Nov 02 10:44:08.102470 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:08.101462    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-9nd6j_51e601b5-d915-4326-b964-523a4f66788d/istio-proxy/0.log"
Nov 02 10:44:08.203596 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:08.203566    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-mqzmk_de093953-9e5f-4d01-9e8f-0404f29e0b82/istio-proxy/0.log"
Nov 02 10:44:08.226370 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:08.226298    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-qwf92_e85060bf-426e-4a87-9e44-3f4577417345/istio-proxy/0.log"
Nov 02 10:44:08.260929 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:08.260871    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-zwbkz_ccf08bc0-8270-410d-a5d4-93be687e910a/istio-proxy/0.log"
Nov 02 10:44:08.299070 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:08.299048    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-75fb58b5b7-xbsfb_3b4a41e8-9183-4a0a-b088-4a292363401f/payload-processing/0.log"
Nov 02 10:44:08.844973 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:08.844942    4449 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cwjgx_1cc56635-aa7a-4702-8575-2c73011a580f/manager/0.log"
Nov 02 10:44:09.208559 ip-10-0-128-217 kubenswrapper[4449]: I1102 10:44:09.208455    4449 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a344ae-7697-4999-938b-776c67f7ed14" path="/var/lib/kubelet/pods/11a344ae-7697-4999-938b-776c67f7ed14/volumes"